diff --git a/Ch10/01_BioBERT_Demo.ipynb b/Ch10/01_BioBERT_Demo.ipynb index 388740d..b9ffce7 100644 --- a/Ch10/01_BioBERT_Demo.ipynb +++ b/Ch10/01_BioBERT_Demo.ipynb @@ -1,1961 +1,1978 @@ { - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "name": "BioBERT_Demo.ipynb", - "provenance": [], - "collapsed_sections": [] - }, - "kernelspec": { - "name": "python3", - "display_name": "Python 3" - }, - "accelerator": "GPU" + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "dnX4TfNmTbeH" + }, + "source": [ + "In this notebook we demostrate how to use BioBERT , which is BERT pre trained on a huge corpus of medical data. We will demonstrate text classification. Tasks such as NER and slot filling can be easily performed by replacing the pre-trained model with the pre-trainied biobert.\n", + "

\n", + "Now to make use of the pre-trained bioBert model with the hugging face transformers library we need the model's weights to be in a form which pyTorch understands. The original model was trained using tensorflow so we need to convert the weights into pyTorch weights. We can then import and use it.\n" + ] }, - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "dnX4TfNmTbeH", - "colab_type": "text" - }, - "source": [ - "In this notebook we demostrate how to use BioBERT , which is BERT pre trained on a huge corpus of medical data. We will demonstrate text classification. Tasks such as NER and slot filling can be easily performed by replacing the pre-trained model with the pre-trainied biobert.\n", - "

\n", - "Now to make use of the pre-trained bioBert model with the hugging face transformers library we need the model's weights to be in a form which pyTorch understands. The original model was trained using tensorflow so we need to convert the weights into pyTorch weights. We can then import and use it.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "mqXcmmf9katE", - "colab_type": "text" - }, - "source": [ - "First we will download the model from github repo of [BioBert](https://github.com/dmis-lab/biobert).
Huge shoutout to this [article](https://medium.com/@acpanjan/download-google-drive-files-using-wget-3c2c025a8b99) which helped in using wget to download the pre-trained model." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "ovmfPZzxi12O", - "colab_type": "code", - "outputId": "efdd7b37-b95b-4f8e-b082-e454675bf886", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 474 - } - }, - "source": [ - "!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD\" -O biobert_weights && rm -rf /tmp/cookies.txt" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "--2020-06-03 08:26:38-- https://docs.google.com/uc?export=download&confirm=c2ta&id=1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD\n", - "Resolving docs.google.com (docs.google.com)... 173.194.76.102, 173.194.76.138, 173.194.76.101, ...\n", - "Connecting to docs.google.com (docs.google.com)|173.194.76.102|:443... connected.\n", - "HTTP request sent, awaiting response... 302 Moved Temporarily\n", - "Location: https://doc-0g-4c-docs.googleusercontent.com/docs/securesc/e2i8cejjg0caiq8kn6rmktr01u0g4ha4/m0smlqaldjg8s3nkj2jpfvg3pohl9a65/1591172775000/13799006341648886493/09375971713260418591Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e=download [following]\n", - "--2020-06-03 08:26:39-- https://doc-0g-4c-docs.googleusercontent.com/docs/securesc/e2i8cejjg0caiq8kn6rmktr01u0g4ha4/m0smlqaldjg8s3nkj2jpfvg3pohl9a65/1591172775000/13799006341648886493/09375971713260418591Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e=download\n", - "Resolving doc-0g-4c-docs.googleusercontent.com (doc-0g-4c-docs.googleusercontent.com)... 74.125.133.132, 2a00:1450:400c:c07::84\n", - "Connecting to doc-0g-4c-docs.googleusercontent.com (doc-0g-4c-docs.googleusercontent.com)|74.125.133.132|:443... connected.\n", - "HTTP request sent, awaiting response... 302 Found\n", - "Location: https://docs.google.com/nonceSigner?nonce=a8l8eti4o0u14&continue=https://doc-0g-4c-docs.googleusercontent.com/docs/securesc/e2i8cejjg0caiq8kn6rmktr01u0g4ha4/m0smlqaldjg8s3nkj2jpfvg3pohl9a65/1591172775000/13799006341648886493/09375971713260418591Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e%3Ddownload&hash=1rgbo8a7k06ud6o1hni33130dk22rh0s [following]\n", - "--2020-06-03 08:26:39-- https://docs.google.com/nonceSigner?nonce=a8l8eti4o0u14&continue=https://doc-0g-4c-docs.googleusercontent.com/docs/securesc/e2i8cejjg0caiq8kn6rmktr01u0g4ha4/m0smlqaldjg8s3nkj2jpfvg3pohl9a65/1591172775000/13799006341648886493/09375971713260418591Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e%3Ddownload&hash=1rgbo8a7k06ud6o1hni33130dk22rh0s\n", - "Connecting to docs.google.com (docs.google.com)|173.194.76.102|:443... connected.\n", - "HTTP request sent, awaiting response... 302 Found\n", - "Location: https://doc-0g-4c-docs.googleusercontent.com/docs/securesc/e2i8cejjg0caiq8kn6rmktr01u0g4ha4/m0smlqaldjg8s3nkj2jpfvg3pohl9a65/1591172775000/13799006341648886493/09375971713260418591Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e=download&nonce=a8l8eti4o0u14&user=09375971713260418591Z&hash=c92psivkvvfl1ighk2v0nvf1o615stct [following]\n", - "--2020-06-03 08:26:39-- https://doc-0g-4c-docs.googleusercontent.com/docs/securesc/e2i8cejjg0caiq8kn6rmktr01u0g4ha4/m0smlqaldjg8s3nkj2jpfvg3pohl9a65/1591172775000/13799006341648886493/09375971713260418591Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e=download&nonce=a8l8eti4o0u14&user=09375971713260418591Z&hash=c92psivkvvfl1ighk2v0nvf1o615stct\n", - "Connecting to doc-0g-4c-docs.googleusercontent.com (doc-0g-4c-docs.googleusercontent.com)|74.125.133.132|:443... connected.\n", - "HTTP request sent, awaiting response... 200 OK\n", - "Length: unspecified [application/x-gzip]\n", - "Saving to: ‘biobert_weights’\n", - "\n", - "biobert_weights [ <=> ] 382.81M 76.3MB/s in 5.0s \n", - "\n", - "2020-06-03 08:26:45 (76.3 MB/s) - ‘biobert_weights’ saved [401403346]\n", - "\n" - ], - "name": "stdout" - } - ] + { + "cell_type": "markdown", + "metadata": { + "id": "mqXcmmf9katE" + }, + "source": [ + "First we will download the model from github repo of [BioBert](https://github.com/dmis-lab/biobert).
Huge shoutout to this [article](https://medium.com/@acpanjan/download-google-drive-files-using-wget-3c2c025a8b99) which helped in using wget to download the pre-trained model." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "6UX0jJep8NRG" + }, + "outputs": [], + "source": [ + "import requests\n", + "\n", + "def download_file_from_google_drive(id, destination):\n", + " URL = \"https://docs.google.com/uc?export=download\"\n", + "\n", + " session = requests.Session()\n", + "\n", + " response = session.get(URL, params = { 'id' : id }, stream = True)\n", + " token = get_confirm_token(response)\n", + "\n", + " if token:\n", + " params = { 'id' : id, 'confirm' : token }\n", + " response = session.get(URL, params = params, stream = True)\n", + "\n", + " save_response_content(response, destination) \n", + "\n", + "def get_confirm_token(response):\n", + " for key, value in response.cookies.items():\n", + " if key.startswith('download_warning'):\n", + " return value\n", + "\n", + " return None\n", + "\n", + "def save_response_content(response, destination):\n", + " CHUNK_SIZE = 32768\n", + "\n", + " with open(destination, \"wb\") as f:\n", + " for chunk in response.iter_content(CHUNK_SIZE):\n", + " if chunk: # filter out keep-alive new chunks\n", + " f.write(chunk)" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "NmeqlPFuJ3xh", + "outputId": "aeae189d-8bb4-419a-e634-81d904178455" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "XtyyyZ-y4nZQ", - "colab_type": "text" - }, - "source": [ - "Install the required libraries" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "--2021-06-18 06:53:30-- https://docs.google.com/uc?export=download&confirm=AN3n&id=1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD\n", + "Resolving docs.google.com (docs.google.com)... 142.250.65.78, 2607:f8b0:4004:810::200e\n", + "Connecting to docs.google.com (docs.google.com)|142.250.65.78|:443... connected.\n", + "HTTP request sent, awaiting response... 302 Moved Temporarily\n", + "Location: https://doc-00-ag-docs.googleusercontent.com/docs/securesc/sna14kp6itmgjq148koh659elc46pjr8/uqeo0ioa1v4caccujf174q4v33ifbhum/1623999150000/13799006341648886493/03088793929649119770Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e=download [following]\n", + "--2021-06-18 06:53:31-- https://doc-00-ag-docs.googleusercontent.com/docs/securesc/sna14kp6itmgjq148koh659elc46pjr8/uqeo0ioa1v4caccujf174q4v33ifbhum/1623999150000/13799006341648886493/03088793929649119770Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e=download\n", + "Resolving doc-00-ag-docs.googleusercontent.com (doc-00-ag-docs.googleusercontent.com)... 142.250.65.65, 2607:f8b0:4004:832::2001\n", + "Connecting to doc-00-ag-docs.googleusercontent.com (doc-00-ag-docs.googleusercontent.com)|142.250.65.65|:443... connected.\n", + "HTTP request sent, awaiting response... 302 Found\n", + "Location: https://docs.google.com/nonceSigner?nonce=koca5q9d7tl6g&continue=https://doc-00-ag-docs.googleusercontent.com/docs/securesc/sna14kp6itmgjq148koh659elc46pjr8/uqeo0ioa1v4caccujf174q4v33ifbhum/1623999150000/13799006341648886493/03088793929649119770Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e%3Ddownload&hash=um3h8bd9ebi8b1j9cp64omte2v5r58f8 [following]\n", + "--2021-06-18 06:53:31-- https://docs.google.com/nonceSigner?nonce=koca5q9d7tl6g&continue=https://doc-00-ag-docs.googleusercontent.com/docs/securesc/sna14kp6itmgjq148koh659elc46pjr8/uqeo0ioa1v4caccujf174q4v33ifbhum/1623999150000/13799006341648886493/03088793929649119770Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e%3Ddownload&hash=um3h8bd9ebi8b1j9cp64omte2v5r58f8\n", + "Connecting to docs.google.com (docs.google.com)|142.250.65.78|:443... connected.\n", + "HTTP request sent, awaiting response... 302 Found\n", + "Location: https://doc-00-ag-docs.googleusercontent.com/docs/securesc/sna14kp6itmgjq148koh659elc46pjr8/uqeo0ioa1v4caccujf174q4v33ifbhum/1623999150000/13799006341648886493/03088793929649119770Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e=download&nonce=koca5q9d7tl6g&user=03088793929649119770Z&hash=muo7k0ephe2pkgfihv91rld5dh31kbb8 [following]\n", + "--2021-06-18 06:53:31-- https://doc-00-ag-docs.googleusercontent.com/docs/securesc/sna14kp6itmgjq148koh659elc46pjr8/uqeo0ioa1v4caccujf174q4v33ifbhum/1623999150000/13799006341648886493/03088793929649119770Z/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD?e=download&nonce=koca5q9d7tl6g&user=03088793929649119770Z&hash=muo7k0ephe2pkgfihv91rld5dh31kbb8\n", + "Connecting to doc-00-ag-docs.googleusercontent.com (doc-00-ag-docs.googleusercontent.com)|142.250.65.65|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: unspecified [application/x-gzip]\n", + "Saving to: ‘biobert_weights’\n", + "\n", + "biobert_weights [ <=> ] 382.81M 138MB/s in 2.8s \n", + "\n", + "2021-06-18 06:53:34 (138 MB/s) - ‘biobert_weights’ saved [401403346]\n", + "\n" + ] + } + ], + "source": [ + "try:\n", + " import google.colab\n", + " !wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD\" -O biobert_weights && rm -rf /tmp/cookies.txt\n", + "except ModuleNotFoundError:\n", + " file_id = '1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD'\n", + " destination = 'Data/biobert_weights'\n", + " download_file_from_google_drive(file_id, destination)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XtyyyZ-y4nZQ" + }, + "source": [ + "Install the required libraries" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "I-2iN0AqjSTD", + "outputId": "d502814e-2c5a-4a99-dc82-c007be2a1941" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "I-2iN0AqjSTD", - "colab_type": "code", - "outputId": "414b45f3-7256-4531-b531-882d45ad8d56", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - } - }, - "source": [ - "!pip install pytorch_transformers\n", - "!pip install transformers" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Collecting pytorch_transformers\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/a3/b7/d3d18008a67e0b968d1ab93ad444fc05699403fa662f634b2f2c318a508b/pytorch_transformers-1.2.0-py3-none-any.whl (176kB)\n", - "\r\u001b[K |█▉ | 10kB 16.4MB/s eta 0:00:01\r\u001b[K |███▊ | 20kB 4.3MB/s eta 0:00:01\r\u001b[K |█████▋ | 30kB 5.6MB/s eta 0:00:01\r\u001b[K |███████▍ | 40kB 5.7MB/s eta 0:00:01\r\u001b[K |█████████▎ | 51kB 4.9MB/s eta 0:00:01\r\u001b[K |███████████▏ | 61kB 5.3MB/s eta 0:00:01\r\u001b[K |█████████████ | 71kB 5.7MB/s eta 0:00:01\r\u001b[K |██████████████▉ | 81kB 6.2MB/s eta 0:00:01\r\u001b[K |████████████████▊ | 92kB 6.5MB/s eta 0:00:01\r\u001b[K |██████████████████▋ | 102kB 6.6MB/s eta 0:00:01\r\u001b[K |████████████████████▍ | 112kB 6.6MB/s eta 0:00:01\r\u001b[K |██████████████████████▎ | 122kB 6.6MB/s eta 0:00:01\r\u001b[K |████████████████████████▏ | 133kB 6.6MB/s eta 0:00:01\r\u001b[K |██████████████████████████ | 143kB 6.6MB/s eta 0:00:01\r\u001b[K |███████████████████████████▉ | 153kB 6.6MB/s eta 0:00:01\r\u001b[K |█████████████████████████████▊ | 163kB 6.6MB/s eta 0:00:01\r\u001b[K |███████████████████████████████▋| 174kB 6.6MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 184kB 6.6MB/s \n", - "\u001b[?25hRequirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (2019.12.20)\n", - "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (1.18.4)\n", - "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (1.13.19)\n", - "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (2.23.0)\n", - "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (1.5.0+cu101)\n", - "Collecting sacremoses\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n", - "\u001b[K |████████████████████████████████| 890kB 12.7MB/s \n", - "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (4.41.1)\n", - "Collecting sentencepiece\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)\n", - "\u001b[K |████████████████████████████████| 1.1MB 36.4MB/s \n", - "\u001b[?25hRequirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_transformers) (0.3.3)\n", - "Requirement already satisfied: botocore<1.17.0,>=1.16.19 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_transformers) (1.16.19)\n", - "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_transformers) (0.10.0)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_transformers) (2020.4.5.1)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_transformers) (3.0.4)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_transformers) (1.24.3)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_transformers) (2.9)\n", - "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch>=1.0.0->pytorch_transformers) (0.16.0)\n", - "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->pytorch_transformers) (1.12.0)\n", - "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->pytorch_transformers) (7.1.2)\n", - "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->pytorch_transformers) (0.15.1)\n", - "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.19->boto3->pytorch_transformers) (0.15.2)\n", - "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.19->boto3->pytorch_transformers) (2.8.1)\n", - "Building wheels for collected packages: sacremoses\n", - " Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893260 sha256=af6043465250e43996a63fb70ee95c58ad6c63eed431b8eb63845c6bf644ee8a\n", - " Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n", - "Successfully built sacremoses\n", - "Installing collected packages: sacremoses, sentencepiece, pytorch-transformers\n", - "Successfully installed pytorch-transformers-1.2.0 sacremoses-0.0.43 sentencepiece-0.1.91\n", - "Collecting transformers\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/48/35/ad2c5b1b8f99feaaf9d7cdadaeef261f098c6e1a6a2935d4d07662a6b780/transformers-2.11.0-py3-none-any.whl (674kB)\n", - "\u001b[K |████████████████████████████████| 675kB 6.5MB/s \n", - "\u001b[?25hRequirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers) (0.0.43)\n", - "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.6/dist-packages (from transformers) (0.1.91)\n", - "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.7)\n", - "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n", - "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)\n", - "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n", - "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.18.4)\n", - "Collecting tokenizers==0.7.0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/14/e5/a26eb4716523808bb0a799fcfdceb6ebf77a18169d9591b2f46a9adb87d9/tokenizers-0.7.0-cp36-cp36m-manylinux1_x86_64.whl (3.8MB)\n", - "\u001b[K |████████████████████████████████| 3.8MB 30.5MB/s \n", - "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)\n", - "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.4)\n", - "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.12.0)\n", - "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n", - "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (0.15.1)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.4.5.1)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.9)\n", - "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n", - "Installing collected packages: tokenizers, transformers\n", - "Successfully installed tokenizers-0.7.0 transformers-2.11.0\n" - ], - "name": "stdout" - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting pytorch_transformers\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/a3/b7/d3d18008a67e0b968d1ab93ad444fc05699403fa662f634b2f2c318a508b/pytorch_transformers-1.2.0-py3-none-any.whl (176kB)\n", + "\u001b[K |████████████████████████████████| 184kB 23.6MB/s \n", + "\u001b[?25hCollecting sentencepiece\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/f5/99/e0808cb947ba10f575839c43e8fafc9cc44e4a7a2c8f79c60db48220a577/sentencepiece-0.1.95-cp37-cp37m-manylinux2014_x86_64.whl (1.2MB)\n", + "\u001b[K |████████████████████████████████| 1.2MB 29.1MB/s \n", + "\u001b[?25hCollecting boto3\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/39/96/5c009100d651f4646639f9777a82777430897590f5fd28bbc077cace09bf/boto3-1.17.97-py2.py3-none-any.whl (131kB)\n", + "\u001b[K |████████████████████████████████| 133kB 46.5MB/s \n", + "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (4.41.1)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (2.23.0)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (1.19.5)\n", + "Collecting sacremoses\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/75/ee/67241dc87f266093c533a2d4d3d69438e57d7a90abb216fa076e7d475d4a/sacremoses-0.0.45-py3-none-any.whl (895kB)\n", + "\u001b[K |████████████████████████████████| 901kB 37.3MB/s \n", + "\u001b[?25hRequirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (2019.12.20)\n", + "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (1.8.1+cu101)\n", + "Collecting jmespath<1.0.0,>=0.7.1\n", + " Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n", + "Collecting s3transfer<0.5.0,>=0.4.0\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/63/d0/693477c688348654ddc21dcdce0817653a294aa43f41771084c25e7ff9c7/s3transfer-0.4.2-py2.py3-none-any.whl (79kB)\n", + "\u001b[K |████████████████████████████████| 81kB 9.7MB/s \n", + "\u001b[?25hCollecting botocore<1.21.0,>=1.20.97\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/94/a2/bd7cd40c61a71c3daede70158b5c0aca918057f63cfbef015c56cccdb6b2/botocore-1.20.97-py2.py3-none-any.whl (7.6MB)\n", + "\u001b[K |████████████████████████████████| 7.6MB 40.9MB/s \n", + "\u001b[?25hRequirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_transformers) (2021.5.30)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_transformers) (3.0.4)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_transformers) (2.10)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_transformers) (1.24.3)\n", + "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->pytorch_transformers) (7.1.2)\n", + "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->pytorch_transformers) (1.15.0)\n", + "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->pytorch_transformers) (1.0.1)\n", + "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.0.0->pytorch_transformers) (3.7.4.3)\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.21.0,>=1.20.97->boto3->pytorch_transformers) (2.8.1)\n", + "\u001b[31mERROR: botocore 1.20.97 has requirement urllib3<1.27,>=1.25.4, but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n", + "Installing collected packages: sentencepiece, jmespath, botocore, s3transfer, boto3, sacremoses, pytorch-transformers\n", + "Successfully installed boto3-1.17.97 botocore-1.20.97 jmespath-0.10.0 pytorch-transformers-1.2.0 s3transfer-0.4.2 sacremoses-0.0.45 sentencepiece-0.1.95\n", + "Collecting transformers\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/00/92/6153f4912b84ee1ab53ab45663d23e7cf3704161cb5ef18b0c07e207cef2/transformers-4.7.0-py3-none-any.whl (2.5MB)\n", + "\u001b[K |████████████████████████████████| 2.5MB 24.6MB/s \n", + "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers) (20.9)\n", + "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.19.5)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from transformers) (3.13)\n", + "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.41.1)\n", + "Collecting huggingface-hub==0.0.8\n", + " Downloading https://files.pythonhosted.org/packages/a1/88/7b1e45720ecf59c6c6737ff332f41c955963090a18e72acbcbeac6b25e86/huggingface_hub-0.0.8-py3-none-any.whl\n", + "Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers) (0.0.45)\n", + "Collecting tokenizers<0.11,>=0.10.1\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d4/e2/df3543e8ffdab68f5acc73f613de9c2b155ac47f162e725dcac87c521c11/tokenizers-0.10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3MB)\n", + "\u001b[K |████████████████████████████████| 3.3MB 40.9MB/s \n", + "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n", + "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.0.12)\n", + "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers) (4.5.0)\n", + "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers) (2.4.7)\n", + "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n", + "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n", + "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.0.1)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2021.5.30)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n", + "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.7.4.3)\n", + "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.4.1)\n", + "Installing collected packages: huggingface-hub, tokenizers, transformers\n", + "Successfully installed huggingface-hub-0.0.8 tokenizers-0.10.3 transformers-4.7.0\n" + ] + } + ], + "source": [ + "!pip install pytorch_transformers\n", + "!pip install transformers" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-4jlHKl--wOb" + }, + "source": [ + "Unzip the file and convert the weights to a pytorch readable form. Stefan-it's solution to the issue [here](https://github.com/huggingface/transformers/issues/457) was extremely helpful." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "6BXvJSsJka0Q", + "outputId": "601808d8-1d06-4acf-bfba-053fcfd1ab4f" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "-4jlHKl--wOb", - "colab_type": "text" - }, - "source": [ - "Unzip the file. And convert the weights to a pytorch readable form. Stefan-it's solution to the issue [here](https://github.com/huggingface/transformers/issues/457) was extremely helpful." - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "bert_config.json\t\t\tmodel.ckpt-1000000.index vocab.txt\n", + "model.ckpt-1000000.data-00000-of-00001\tmodel.ckpt-1000000.meta\n" + ] + } + ], + "source": [ + "try:\n", + "\n", + " import google.colab\n", + " !tar -xzf biobert_weights\n", + " !ls biobert_v1.1_pubmed/\n", + "\n", + "except ModuleNotFoundError:\n", + "\n", + " import tarfile\n", + " tar = tarfile.open('Data/biobert_weights','r')\n", + " tar.extractall('Data/')\n", + " tar.close()\n", + "\n", + " import os\n", + " print(os.listdir('Data/biobert_v1.1_pubmed'))" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "id": "4ErQdDnYraNJ" + }, + "outputs": [], + "source": [ + "PATH = \".\"\n", + "try:\n", + " import google.colab\n", + " PATH = './'\n", + "except ModuleNotFoundError:\n", + " PATH = 'Data/'" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "37djYqLV5NBb", + "outputId": "9d2992f6-c891-434d-f74f-363b54088376" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "6BXvJSsJka0Q", - "colab_type": "code", - "outputId": "c3f4b9fd-aa40-45b8-9801-eead6bcfd7d2", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 54 - } - }, - "source": [ - "!tar -xzf biobert_weights\n", - "!ls biobert_v1.1_pubmed/" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "bert_config.json\t\t\tmodel.ckpt-1000000.index vocab.txt\n", - "model.ckpt-1000000.data-00000-of-00001\tmodel.ckpt-1000000.meta\n" - ], - "name": "stdout" - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "2021-06-18 06:54:00.045437: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0\n", + "Building PyTorch model from configuration: BertConfig {\n", + " \"attention_probs_dropout_prob\": 0.1,\n", + " \"gradient_checkpointing\": false,\n", + " \"hidden_act\": \"gelu\",\n", + " \"hidden_dropout_prob\": 0.1,\n", + " \"hidden_size\": 768,\n", + " \"initializer_range\": 0.02,\n", + " \"intermediate_size\": 3072,\n", + " \"layer_norm_eps\": 1e-12,\n", + " \"max_position_embeddings\": 512,\n", + " \"model_type\": \"bert\",\n", + " \"num_attention_heads\": 12,\n", + " \"num_hidden_layers\": 12,\n", + " \"pad_token_id\": 0,\n", + " \"position_embedding_type\": \"absolute\",\n", + " \"transformers_version\": \"4.7.0\",\n", + " \"type_vocab_size\": 2,\n", + " \"use_cache\": true,\n", + " \"vocab_size\": 28996\n", + "}\n", + "\n", + "Converting TensorFlow checkpoint from /content/biobert_v1.1_pubmed/model.ckpt-1000000\n", + "Loading TF weight bert/embeddings/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/embeddings/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/embeddings/position_embeddings with shape [512, 768]\n", + "Loading TF weight bert/embeddings/token_type_embeddings with shape [2, 768]\n", + "Loading TF weight bert/embeddings/word_embeddings with shape [28996, 768]\n", + "Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_0/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_0/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_0/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_0/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_0/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_0/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_0/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_0/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_0/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_0/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_1/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_1/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_1/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_1/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_1/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_1/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_1/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_1/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_1/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_1/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_10/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_10/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_10/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_10/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_10/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_10/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_10/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_10/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_10/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_10/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_11/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_11/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_11/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_11/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_11/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_11/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_11/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_11/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_11/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_11/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_2/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_2/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_2/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_2/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_2/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_2/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_2/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_2/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_2/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_2/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_3/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_3/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_3/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_3/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_3/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_3/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_3/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_3/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_3/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_3/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_4/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_4/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_4/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_4/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_4/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_4/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_4/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_4/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_4/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_4/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_5/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_5/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_5/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_5/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_5/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_5/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_5/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_5/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_5/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_5/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_6/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_6/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_6/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_6/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_6/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_6/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_6/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_6/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_6/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_6/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_7/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_7/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_7/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_7/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_7/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_7/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_7/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_7/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_7/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_7/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_8/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_8/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_8/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_8/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_8/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_8/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_8/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_8/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_8/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_8/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_9/attention/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_9/attention/self/key/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_9/attention/self/key/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_9/attention/self/query/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_9/attention/self/query/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_9/attention/self/value/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_9/attention/self/value/kernel with shape [768, 768]\n", + "Loading TF weight bert/encoder/layer_9/intermediate/dense/bias with shape [3072]\n", + "Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel with shape [768, 3072]\n", + "Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta with shape [768]\n", + "Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma with shape [768]\n", + "Loading TF weight bert/encoder/layer_9/output/dense/bias with shape [768]\n", + "Loading TF weight bert/encoder/layer_9/output/dense/kernel with shape [3072, 768]\n", + "Loading TF weight bert/pooler/dense/bias with shape [768]\n", + "Loading TF weight bert/pooler/dense/kernel with shape [768, 768]\n", + "Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'embeddings', 'position_embeddings']\n", + "Initialize PyTorch weight ['bert', 'embeddings', 'token_type_embeddings']\n", + "Initialize PyTorch weight ['bert', 'embeddings', 'word_embeddings']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'key', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'key', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'query', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'query', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'value', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'value', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'intermediate', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'intermediate', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'LayerNorm', 'beta']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'LayerNorm', 'gamma']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'kernel']\n", + "Initialize PyTorch weight ['bert', 'pooler', 'dense', 'bias']\n", + "Initialize PyTorch weight ['bert', 'pooler', 'dense', 'kernel']\n", + "Save PyTorch model to biobert_v1.1_pubmed/pytorch_model.bin\n" + ] + } + ], + "source": [ + "try:\n", + " import google.colab\n", + " \n", + " !transformers-cli convert --model_type bert --tf_checkpoint biobert_v1.1_pubmed/model.ckpt-1000000 --config biobert_v1.1_pubmed/bert_config.json --pytorch_dump_output biobert_v1.1_pubmed/pytorch_model.bin\n", + "\n", + "except ModuleNotFoundError:\n", + " \n", + " !transformers-cli convert --model_type bert --tf_checkpoint Data/biobert_v1.1_pubmed/model.ckpt-1000000 --config Data/biobert_v1.1_pubmed/bert_config.json --pytorch_dump_output Data/biobert_v1.1_pubmed/pytorch_model.bin" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "_qgMNMnWpwLO", + "outputId": "4309d0c7-58ed-4a3b-ad8f-48ed4db3e438" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "37djYqLV5NBb", - "colab_type": "code", - "outputId": "20ff8aa9-e6a2-4f7b-87af-9a3c978cc658", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - } - }, - "source": [ - "!transformers-cli convert --model_type bert --tf_checkpoint biobert_v1.1_pubmed/model.ckpt-1000000 --config biobert_v1.1_pubmed/bert_config.json --pytorch_dump_output biobert_v1.1_pubmed/pytorch_model.bin" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "2020-06-03 08:28:26.416665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n", - "Building PyTorch model from configuration: BertConfig {\n", - " \"attention_probs_dropout_prob\": 0.1,\n", - " \"hidden_act\": \"gelu\",\n", - " \"hidden_dropout_prob\": 0.1,\n", - " \"hidden_size\": 768,\n", - " \"initializer_range\": 0.02,\n", - " \"intermediate_size\": 3072,\n", - " \"layer_norm_eps\": 1e-12,\n", - " \"max_position_embeddings\": 512,\n", - " \"model_type\": \"bert\",\n", - " \"num_attention_heads\": 12,\n", - " \"num_hidden_layers\": 12,\n", - " \"pad_token_id\": 0,\n", - " \"type_vocab_size\": 2,\n", - " \"vocab_size\": 28996\n", - "}\n", - "\n", - "INFO:transformers.modeling_bert:Converting TensorFlow checkpoint from /content/biobert_v1.1_pubmed/model.ckpt-1000000\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/embeddings/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/embeddings/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/embeddings/position_embeddings with shape [512, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/embeddings/token_type_embeddings with shape [2, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/embeddings/word_embeddings with shape [28996, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_0/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_1/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_10/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_11/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_2/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_3/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_4/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_5/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_6/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_7/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_8/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/self/key/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/self/key/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/self/query/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/self/query/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/self/value/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/attention/self/value/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/intermediate/dense/bias with shape [3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel with shape [768, 3072]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/output/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/encoder/layer_9/output/dense/kernel with shape [3072, 768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/pooler/dense/bias with shape [768]\n", - "INFO:transformers.modeling_bert:Loading TF weight bert/pooler/dense/kernel with shape [768, 768]\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'embeddings', 'position_embeddings']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'embeddings', 'token_type_embeddings']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'embeddings', 'word_embeddings']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'key', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'key', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'query', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'query', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'value', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'value', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'intermediate', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'intermediate', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'LayerNorm', 'beta']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'LayerNorm', 'gamma']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'kernel']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'pooler', 'dense', 'bias']\n", - "INFO:transformers.modeling_bert:Initialize PyTorch weight ['bert', 'pooler', 'dense', 'kernel']\n", - "Save PyTorch model to biobert_v1.1_pubmed/pytorch_model.bin\n" - ], - "name": "stdout" - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "bert_config.json\t\t\tmodel.ckpt-1000000.meta\n", + "model.ckpt-1000000.data-00000-of-00001\tpytorch_model.bin\n", + "model.ckpt-1000000.index\t\tvocab.txt\n", + "config.json\t\t\t\tmodel.ckpt-1000000.meta\n", + "model.ckpt-1000000.data-00000-of-00001\tpytorch_model.bin\n", + "model.ckpt-1000000.index\t\tvocab.txt\n" + ] + } + ], + "source": [ + "try:\n", + " import google.colab \n", + " !ls biobert_v1.1_pubmed/\n", + " !mv biobert_v1.1_pubmed/bert_config.json biobert_v1.1_pubmed/config.json\n", + " !ls biobert_v1.1_pubmed/\n", + "except ModuleNotFoundError:\n", + " import os\n", + " import shutil\n", + " print(os.listdir(PATH+'biobert_v1.1_pubmed'))\n", + " shutil.copy(PATH+'biobert_v1.1_pubmed/bert_config.json',PATH+'biobert_v1.1_pubmed/config.json')\n", + " print(os.listdir(PATH+'biobert_v1.1_pubmed'))" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "id": "Vb8sPt0M6Tn-" + }, + "outputs": [], + "source": [ + "from pytorch_transformers import BertModel\n", + "model = BertModel.from_pretrained(PATH+'biobert_v1.1_pubmed')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FEheik_HlMUS" + }, + "source": [ + "Download the mtsamples.csv dataset which can be found [here](https://www.kaggle.com/tboyle10/medicaltranscriptions)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 72, + "resources": { + "http://localhost:8080/nbextensions/google.colab/files.js": { + "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK", + "headers": [ + [ + "content-type", + "application/javascript" + ] + ], + "ok": true, + "status": 200, + "status_text": "" + } + } }, + "id": "W8I3KD0a9ttJ", + "outputId": "3f8c316e-3143-45e3-bc12-95fe64fb0041" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "pmep39JV51uQ", - "colab_type": "code", - "outputId": "631b2c49-fa2c-42a5-9287-871b59357a66", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 126 - } - }, - "source": [ - "!ls biobert_v1.1_pubmed/\n", - "!mv biobert_v1.1_pubmed/bert_config.json biobert_v1.1_pubmed/config.json\n", - "!ls biobert_v1.1_pubmed/" + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " Upload widget is only available when the cell has been executed in the\n", + " current browser session. Please rerun this cell to enable.\n", + " \n", + " " ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "bert_config.json\t\t\tmodel.ckpt-1000000.meta\n", - "model.ckpt-1000000.data-00000-of-00001\tpytorch_model.bin\n", - "model.ckpt-1000000.index\t\tvocab.txt\n", - "config.json\t\t\t\tmodel.ckpt-1000000.meta\n", - "model.ckpt-1000000.data-00000-of-00001\tpytorch_model.bin\n", - "model.ckpt-1000000.index\t\tvocab.txt\n" - ], - "name": "stdout" - } + "text/plain": [ + "" ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" }, { - "cell_type": "code", - "metadata": { - "id": "Vb8sPt0M6Tn-", - "colab_type": "code", - "colab": {} - }, - "source": [ - "from pytorch_transformers import BertModel\n", - "model = BertModel.from_pretrained('biobert_v1.1_pubmed')" - ], - "execution_count": 0, - "outputs": [] + "name": "stdout", + "output_type": "stream", + "text": [ + "Saving mtsamples.csv to mtsamples.csv\n" + ] + } + ], + "source": [ + "#importing the dataset\n", + "try:\n", + " from google.colab import files\n", + " uploaded = files.upload()\n", + "except ModuleNotFoundError:\n", + " print('Not Using Colab')" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 306 }, + "id": "r2bl_Z96IPxN", + "outputId": "80c65d3d-28e8-4b53-e6f6-c94822a7ae15" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "W8I3KD0a9ttJ", - "colab_type": "code", - "outputId": "201c4649-2aaa-44fd-c060-7865ae8d21b2", - "colab": { - "resources": { - "http://localhost:8080/nbextensions/google.colab/files.js": { - "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=", - "ok": true, - "headers": [ - [ - "content-type", - "application/javascript" - ] - ], - "status": 200, - "status_text": "" - } - }, - "base_uri": "https://localhost:8080/", - "height": 57 - } - }, - "source": [ - "#importing the dataset\n", - "from google.colab import files\n", - "uploaded = files.upload()" + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Unnamed: 0descriptionmedical_specialtysample_nametranscriptionkeywords
00A 23-year-old white female presents with comp...Allergy / ImmunologyAllergic RhinitisSUBJECTIVE:, This 23-year-old white female pr...allergy / immunology, allergic rhinitis, aller...
11Consult for laparoscopic gastric bypass.BariatricsLaparoscopic Gastric Bypass Consult - 2PAST MEDICAL HISTORY:, He has difficulty climb...bariatrics, laparoscopic gastric bypass, weigh...
22Consult for laparoscopic gastric bypass.BariatricsLaparoscopic Gastric Bypass Consult - 1HISTORY OF PRESENT ILLNESS: , I have seen ABC ...bariatrics, laparoscopic gastric bypass, heart...
332-D M-Mode. Doppler.Cardiovascular / Pulmonary2-D Echocardiogram - 12-D M-MODE: , ,1. Left atrial enlargement wit...cardiovascular / pulmonary, 2-d m-mode, dopple...
442-D EchocardiogramCardiovascular / Pulmonary2-D Echocardiogram - 21. The left ventricular cavity size and wall ...cardiovascular / pulmonary, 2-d, doppler, echo...
\n", + "
" ], - "execution_count": 0, - "outputs": [ - { - "output_type": "display_data", - "data": { - "text/html": [ - "\n", - " \n", - " \n", - " Upload widget is only available when the cell has been executed in the\n", - " current browser session. Please rerun this cell to enable.\n", - " \n", - " " - ], - "text/plain": [ - "" - ] - }, - "metadata": { - "tags": [] - } - }, - { - "output_type": "stream", - "text": [ - "Saving 64826_127612_bundle_archive.zip to 64826_127612_bundle_archive.zip\n" - ], - "name": "stdout" - } + "text/plain": [ + " Unnamed: 0 ... keywords\n", + "0 0 ... allergy / immunology, allergic rhinitis, aller...\n", + "1 1 ... bariatrics, laparoscopic gastric bypass, weigh...\n", + "2 2 ... bariatrics, laparoscopic gastric bypass, heart...\n", + "3 3 ... cardiovascular / pulmonary, 2-d m-mode, dopple...\n", + "4 4 ... cardiovascular / pulmonary, 2-d, doppler, echo...\n", + "\n", + "[5 rows x 6 columns]" ] + }, + "execution_count": 10, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "import pandas as pd\n", + "df = pd.read_csv(PATH+'mtsamples.csv')\n", + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "iQXgD_aMNHs5", + "outputId": "0f0f90de-bcee-4cca-9e16-b2162468fe76" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "e690UL-mEcGa", - "colab_type": "code", - "outputId": "5e72d882-20db-42f2-f159-972e103d3bbb", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 54 - } - }, - "source": [ - "!unzip 64826_127612_bundle_archive.zip" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Archive: 64826_127612_bundle_archive.zip\n", - " inflating: mtsamples.csv \n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "NApNEpWAEmE7", - "colab_type": "code", - "outputId": "02f5e055-e447-470d-881a-1ae1d07245f3", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 54 - } - }, - "source": [ - "!ls" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "64826_127612_bundle_archive.zip biobert_weights sample_data\n", - "biobert_v1.1_pubmed\t\t mtsamples.csv\n" - ], - "name": "stdout" - } + "data": { + "text/plain": [ + "(4999, 6)" ] + }, + "execution_count": 11, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "df.shape" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "rp0OmaS8IcQ-", + "outputId": "2948cdc5-dbe9-4935-927c-11df583975e1" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "FEheik_HlMUS", - "colab_type": "text" - }, - "source": [ - "Load the dataset which can be found [here](https://www.kaggle.com/tboyle10/medicaltranscriptions)" + "data": { + "text/plain": [ + " Surgery 1103\n", + " Consult - History and Phy. 516\n", + " Cardiovascular / Pulmonary 372\n", + " Orthopedic 355\n", + " Radiology 273\n", + " General Medicine 259\n", + " Gastroenterology 230\n", + " Neurology 223\n", + " SOAP / Chart / Progress Notes 166\n", + " Obstetrics / Gynecology 160\n", + " Urology 158\n", + " Discharge Summary 108\n", + " ENT - Otolaryngology 98\n", + " Neurosurgery 94\n", + " Hematology - Oncology 90\n", + " Ophthalmology 83\n", + " Nephrology 81\n", + " Emergency Room Reports 75\n", + " Pediatrics - Neonatal 70\n", + " Pain Management 62\n", + " Psychiatry / Psychology 53\n", + " Office Notes 51\n", + " Podiatry 47\n", + " Dermatology 29\n", + " Cosmetic / Plastic Surgery 27\n", + " Dentistry 27\n", + " Letters 23\n", + " Physical Medicine - Rehab 21\n", + " Sleep Medicine 20\n", + " Endocrinology 19\n", + " Bariatrics 18\n", + " IME-QME-Work Comp etc. 16\n", + " Chiropractic 14\n", + " Rheumatology 10\n", + " Diets and Nutritions 10\n", + " Speech - Language 9\n", + " Lab Medicine - Pathology 8\n", + " Autopsy 8\n", + " Allergy / Immunology 7\n", + " Hospice - Palliative Care 6\n", + "Name: medical_specialty, dtype: int64" ] + }, + "execution_count": 12, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "df['medical_specialty'].value_counts()" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "id": "NblyeQCsIrLY" + }, + "outputs": [], + "source": [ + "#lets try predicting the medical_speciality from the description\n", + "#dataset highly imbalanced. Could remove ones less which are less than 5%. But as it is a demonstration let us just proceed with how the dataset it and look at the results" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EnS0NLFRo-Ag" + }, + "source": [ + "From here it will be he same as the IMDB_sentiment_classification (ch4) notebook so we will not re-explain every step." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "sy_R-uviOzxI", + "outputId": "8aab73cc-3786-40f8-b4af-f44dc77ad453" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "r2bl_Z96IPxN", - "colab_type": "code", - "outputId": "3f5fb9c4-fe55-4530-b45f-6cdeb42cd9b7", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 198 - } - }, - "source": [ - "import pandas as pd\n", - "df = pd.read_csv('mtsamples.csv')\n", - "df.head()" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
Unnamed: 0descriptionmedical_specialtysample_nametranscriptionkeywords
00A 23-year-old white female presents with comp...Allergy / ImmunologyAllergic RhinitisSUBJECTIVE:, This 23-year-old white female pr...allergy / immunology, allergic rhinitis, aller...
11Consult for laparoscopic gastric bypass.BariatricsLaparoscopic Gastric Bypass Consult - 2PAST MEDICAL HISTORY:, He has difficulty climb...bariatrics, laparoscopic gastric bypass, weigh...
22Consult for laparoscopic gastric bypass.BariatricsLaparoscopic Gastric Bypass Consult - 1HISTORY OF PRESENT ILLNESS: , I have seen ABC ...bariatrics, laparoscopic gastric bypass, heart...
332-D M-Mode. Doppler.Cardiovascular / Pulmonary2-D Echocardiogram - 12-D M-MODE: , ,1. Left atrial enlargement wit...cardiovascular / pulmonary, 2-d m-mode, dopple...
442-D EchocardiogramCardiovascular / Pulmonary2-D Echocardiogram - 21. The left ventricular cavity size and wall ...cardiovascular / pulmonary, 2-d, doppler, echo...
\n", - "
" - ], - "text/plain": [ - " Unnamed: 0 ... keywords\n", - "0 0 ... allergy / immunology, allergic rhinitis, aller...\n", - "1 1 ... bariatrics, laparoscopic gastric bypass, weigh...\n", - "2 2 ... bariatrics, laparoscopic gastric bypass, heart...\n", - "3 3 ... cardiovascular / pulmonary, 2-d m-mode, dopple...\n", - "4 4 ... cardiovascular / pulmonary, 2-d, doppler, echo...\n", - "\n", - "[5 rows x 6 columns]" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 38 - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting pytorch_pretrained_bert\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d7/e0/c08d5553b89973d9a240605b9c12404bcf8227590de62bae27acbcfe076b/pytorch_pretrained_bert-0.6.2-py3-none-any.whl (123kB)\n", + "\r", + "\u001b[K |██▋ | 10kB 17.2MB/s eta 0:00:01\r", + "\u001b[K |█████▎ | 20kB 23.1MB/s eta 0:00:01\r", + "\u001b[K |████████ | 30kB 25.9MB/s eta 0:00:01\r", + "\u001b[K |██████████▋ | 40kB 21.8MB/s eta 0:00:01\r", + "\u001b[K |█████████████▎ | 51kB 19.6MB/s eta 0:00:01\r", + "\u001b[K |███████████████▉ | 61kB 21.3MB/s eta 0:00:01\r", + "\u001b[K |██████████████████▌ | 71kB 19.8MB/s eta 0:00:01\r", + "\u001b[K |█████████████████████▏ | 81kB 20.5MB/s eta 0:00:01\r", + "\u001b[K |███████████████████████▉ | 92kB 21.3MB/s eta 0:00:01\r", + "\u001b[K |██████████████████████████▌ | 102kB 22.5MB/s eta 0:00:01\r", + "\u001b[K |█████████████████████████████▏ | 112kB 22.5MB/s eta 0:00:01\r", + "\u001b[K |███████████████████████████████▊| 122kB 22.5MB/s eta 0:00:01\r", + "\u001b[K |████████████████████████████████| 133kB 22.5MB/s \n", + "\u001b[?25hCollecting pytorch-nlp\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/4f/51/f0ee1efb75f7cc2e3065c5da1363d6be2eec79691b2821594f3f2329528c/pytorch_nlp-0.5.0-py3-none-any.whl (90kB)\n", + "\u001b[K |████████████████████████████████| 92kB 9.6MB/s \n", + "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (4.41.1)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (1.19.5)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (2.23.0)\n", + "Requirement already satisfied: boto3 in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (1.17.97)\n", + "Requirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (1.8.1+cu101)\n", + "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (2019.12.20)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_pretrained_bert) (3.0.4)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_pretrained_bert) (2.10)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_pretrained_bert) (1.24.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_pretrained_bert) (2021.5.30)\n", + "Requirement already satisfied: botocore<1.21.0,>=1.20.97 in /usr/local/lib/python3.7/dist-packages (from boto3->pytorch_pretrained_bert) (1.20.97)\n", + "Requirement already satisfied: s3transfer<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from boto3->pytorch_pretrained_bert) (0.4.2)\n", + "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from boto3->pytorch_pretrained_bert) (0.10.0)\n", + "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=0.4.1->pytorch_pretrained_bert) (3.7.4.3)\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.21.0,>=1.20.97->boto3->pytorch_pretrained_bert) (2.8.1)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.21.0,>=1.20.97->boto3->pytorch_pretrained_bert) (1.15.0)\n", + "Installing collected packages: pytorch-pretrained-bert, pytorch-nlp\n", + "Successfully installed pytorch-nlp-0.5.0 pytorch-pretrained-bert-0.6.2\n" + ] + } + ], + "source": [ + "#importing a few necessary packages and setting the DATA directory\n", + "DATA_DIR=\".\"\n", + "import os\n", + "import numpy as np\n", + "import pickle\n", + "import tensorflow as tf\n", + "import warnings\n", + "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n", + "\n", + "# install BERT\n", + "!pip install pytorch_pretrained_bert pytorch-nlp\n", + "\n", + "# BERT imports\n", + "import torch\n", + "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n", + "from keras.preprocessing.sequence import pad_sequences\n", + "from sklearn.model_selection import train_test_split\n", + "from pytorch_pretrained_bert import BertTokenizer, BertConfig\n", + "from pytorch_pretrained_bert import BertAdam, BertForSequenceClassification\n", + "from tqdm import tqdm, trange\n", + "import pandas as pd\n", + "import io\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "# specify GPU device\n", + "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", + "if torch.cuda.is_available():\n", + " n_gpu = torch.cuda.device_count()\n", + " torch.cuda.get_device_name(0)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 306 }, + "id": "WZvP-pIySUXH", + "outputId": "9f39ecf7-cafd-4f9e-edd4-003350a070c1" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "iQXgD_aMNHs5", - "colab_type": "code", - "outputId": "4c906192-cf79-4eb4-dae1-411eb80b4c76", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 35 - } - }, - "source": [ - "df.shape" + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Unnamed: 0descriptionmedical_specialtysample_nametranscriptionkeywords
00A 23-year-old white female presents with comp...0Allergic RhinitisSUBJECTIVE:, This 23-year-old white female pr...allergy / immunology, allergic rhinitis, aller...
11Consult for laparoscopic gastric bypass.2Laparoscopic Gastric Bypass Consult - 2PAST MEDICAL HISTORY:, He has difficulty climb...bariatrics, laparoscopic gastric bypass, weigh...
22Consult for laparoscopic gastric bypass.2Laparoscopic Gastric Bypass Consult - 1HISTORY OF PRESENT ILLNESS: , I have seen ABC ...bariatrics, laparoscopic gastric bypass, heart...
332-D M-Mode. Doppler.32-D Echocardiogram - 12-D M-MODE: , ,1. Left atrial enlargement wit...cardiovascular / pulmonary, 2-d m-mode, dopple...
442-D Echocardiogram32-D Echocardiogram - 21. The left ventricular cavity size and wall ...cardiovascular / pulmonary, 2-d, doppler, echo...
\n", + "
" ], - "execution_count": 0, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "(4999, 6)" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 39 - } + "text/plain": [ + " Unnamed: 0 ... keywords\n", + "0 0 ... allergy / immunology, allergic rhinitis, aller...\n", + "1 1 ... bariatrics, laparoscopic gastric bypass, weigh...\n", + "2 2 ... bariatrics, laparoscopic gastric bypass, heart...\n", + "3 3 ... cardiovascular / pulmonary, 2-d m-mode, dopple...\n", + "4 4 ... cardiovascular / pulmonary, 2-d, doppler, echo...\n", + "\n", + "[5 rows x 6 columns]" ] + }, + "execution_count": 15, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "from sklearn.preprocessing import LabelEncoder\n", + "\n", + "#sentiment is positive and negative we need to convert it to 0,1\n", + "le = LabelEncoder()\n", + "df[\"medical_specialty\"] = le.fit_transform(df[\"medical_specialty\"])\n", + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "UGJEUBCrTQJE", + "outputId": "c3276a5d-9afb-4252-9702-9b130aca1b70" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "rp0OmaS8IcQ-", - "colab_type": "code", - "outputId": "3b4e435b-9843-4af2-afef-934537c0ad54", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 763 - } - }, - "source": [ - "df['medical_specialty'].value_counts()" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - " Surgery 1103\n", - " Consult - History and Phy. 516\n", - " Cardiovascular / Pulmonary 372\n", - " Orthopedic 355\n", - " Radiology 273\n", - " General Medicine 259\n", - " Gastroenterology 230\n", - " Neurology 223\n", - " SOAP / Chart / Progress Notes 166\n", - " Obstetrics / Gynecology 160\n", - " Urology 158\n", - " Discharge Summary 108\n", - " ENT - Otolaryngology 98\n", - " Neurosurgery 94\n", - " Hematology - Oncology 90\n", - " Ophthalmology 83\n", - " Nephrology 81\n", - " Emergency Room Reports 75\n", - " Pediatrics - Neonatal 70\n", - " Pain Management 62\n", - " Psychiatry / Psychology 53\n", - " Office Notes 51\n", - " Podiatry 47\n", - " Dermatology 29\n", - " Cosmetic / Plastic Surgery 27\n", - " Dentistry 27\n", - " Letters 23\n", - " Physical Medicine - Rehab 21\n", - " Sleep Medicine 20\n", - " Endocrinology 19\n", - " Bariatrics 18\n", - " IME-QME-Work Comp etc. 16\n", - " Chiropractic 14\n", - " Diets and Nutritions 10\n", - " Rheumatology 10\n", - " Speech - Language 9\n", - " Autopsy 8\n", - " Lab Medicine - Pathology 8\n", - " Allergy / Immunology 7\n", - " Hospice - Palliative Care 6\n", - "Name: medical_specialty, dtype: int64" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 40 - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Tokenize the first sentence:\n", + "['[CLS]', 'a', '23', '-', 'year', '-', 'old', 'white', 'female', 'presents', 'with', 'complaint', 'of', 'all', '##er', '##gies', '.', '[SEP]']\n" + ] + } + ], + "source": [ + "description = list(df['description'])\n", + "# Tokenize with BERT tokenizer\n", + "tokenizer = BertTokenizer.from_pretrained(PATH+'biobert_v1.1_pubmed', do_lower_case=True)\n", + "\n", + "\n", + "# Restricting the max size of Tokens to 512(BERT doest accept any more than this)\n", + "tokenized_texts = list(map(lambda t: ['[CLS]']+tokenizer.tokenize(t)+['[SEP]'] , description))\n", + "print (\"Tokenize the first sentence:\")\n", + "print (tokenized_texts[0])" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "id": "9oNK7hbPT0C6" + }, + "outputs": [], + "source": [ + "classes = list(df['medical_specialty'])" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "id": "VPgT9lnYUJ_V" + }, + "outputs": [], + "source": [ + "# Set the maximum sequence length. \n", + "MAX_LEN = 128\n", + "\n", + "# Pad our input tokens so that everything has a uniform length\n", + "input_ids = pad_sequences(list(map(tokenizer.convert_tokens_to_ids, tokenized_texts)),\n", + " maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "id": "-cRpNSYDUWKZ" + }, + "outputs": [], + "source": [ + "# Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary\n", + "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]\n", + "input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "id": "aP95AoULUloD" + }, + "outputs": [], + "source": [ + "# Create attention masks\n", + "attention_masks = []\n", + "# Create a mask of 1s for each token followed by 0s for padding\n", + "for seq in input_ids:\n", + " seq_mask = [float(i>0) for i in seq]\n", + " attention_masks.append(seq_mask)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "id": "qauU8icyUngJ" + }, + "outputs": [], + "source": [ + "batch_size = 16\n", + "\n", + "# Use train_test_split to split our data into train and validation sets for training\n", + "train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(input_ids, classes, \n", + " random_state=2020, test_size=0.1)\n", + "train_masks, validation_masks, _, _ = train_test_split(attention_masks, input_ids,\n", + " random_state=2020, test_size=0.1)\n", + " \n", + "# Convert all of our data into torch tensors, the required datatype for our model\n", + "train_inputs = torch.tensor(train_inputs)\n", + "validation_inputs = torch.tensor(validation_inputs)\n", + "train_labels = torch.tensor(train_labels)\n", + "validation_labels = torch.tensor(validation_labels)\n", + "train_masks = torch.tensor(train_masks)\n", + "validation_masks = torch.tensor(validation_masks)\n", + "\n", + "# Create an iterator of our data with torch DataLoader \n", + "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n", + "train_sampler = RandomSampler(train_data)\n", + "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n", + "validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\n", + "validation_sampler = SequentialSampler(validation_data)\n", + "validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "ydBJogc4Uwdl", + "outputId": "a27fa011-41fc-4662-f231-af9f016eadd5" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "NblyeQCsIrLY", - "colab_type": "code", - "colab": {} - }, - "source": [ - "#lets try predicting the medical_speciality from the description\n", - "#dataset highly imbalanced. Could remove ones less which are less than 5%. But as it is a demonstratoin let us just proceed with how the dataset it and look at the results" - ], - "execution_count": 0, - "outputs": [] + "name": "stdout", + "output_type": "stream", + "text": [ + "BertForSequenceClassification(\n", + " (bert): BertModel(\n", + " (embeddings): BertEmbeddings(\n", + " (word_embeddings): Embedding(28996, 768, padding_idx=0)\n", + " (position_embeddings): Embedding(512, 768)\n", + " (token_type_embeddings): Embedding(2, 768)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (encoder): BertEncoder(\n", + " (layer): ModuleList(\n", + " (0): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (1): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (2): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (3): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (4): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (5): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (6): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (7): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (8): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (9): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (10): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (11): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " )\n", + " )\n", + " (pooler): BertPooler(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (activation): Tanh()\n", + " )\n", + " )\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (classifier): Linear(in_features=768, out_features=40, bias=True)\n", + ")\n" + ] + } + ], + "source": [ + "#Loading pre trained BERT\n", + "model = BertForSequenceClassification.from_pretrained(PATH+'biobert_v1.1_pubmed', num_labels=40)#binary classification\n", + "\n", + "if torch.cuda.is_available():\n", + " print(model.cuda())\n", + "else:\n", + " print(model)" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 788 }, + "id": "RzGq3XXmU37D", + "outputId": "113452a0-dd5f-4b34-f30d-1ec6d1a2e652" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "EnS0NLFRo-Ag", - "colab_type": "text" - }, - "source": [ - "From here it will be he same as the IMDB_sentiment_classification (ch4) notebook so we will not re-explain every step." - ] + "name": "stderr", + "output_type": "stream", + "text": [ + "t_total value of -1 results in schedule not being applied\n", + "Epoch: 0%| | 0/4 [00:00=0.4.1 in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (1.5.0+cu101)\n", - "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (1.13.19)\n", - "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (1.18.4)\n", - "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (2.23.0)\n", - "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (4.41.1)\n", - "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch>=0.4.1->pytorch_pretrained_bert) (0.16.0)\n", - "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_pretrained_bert) (0.3.3)\n", - "Requirement already satisfied: botocore<1.17.0,>=1.16.19 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_pretrained_bert) (1.16.19)\n", - "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_pretrained_bert) (0.10.0)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_pretrained_bert) (1.24.3)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_pretrained_bert) (2020.4.5.1)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_pretrained_bert) (3.0.4)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_pretrained_bert) (2.9)\n", - "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.19->boto3->pytorch_pretrained_bert) (0.15.2)\n", - "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.19->boto3->pytorch_pretrained_bert) (2.8.1)\n", - "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.17.0,>=1.16.19->boto3->pytorch_pretrained_bert) (1.12.0)\n" - ], - "name": "stdout" - }, - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "'Tesla P100-PCIE-16GB'" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 42 - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Train loss: 2.248300828832261\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "WZvP-pIySUXH", - "colab_type": "code", - "outputId": "406a181f-8101-4029-d3f4-e0c7ea4b0c0d", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 198 - } - }, - "source": [ - "from sklearn.preprocessing import LabelEncoder\n", - "\n", - "#sentiment is positive and negative we need to convert it to 0,1\n", - "le = LabelEncoder()\n", - "df[\"medical_specialty\"] = le.fit_transform(df[\"medical_specialty\"])\n", - "df.head()" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
Unnamed: 0descriptionmedical_specialtysample_nametranscriptionkeywords
00A 23-year-old white female presents with comp...0Allergic RhinitisSUBJECTIVE:, This 23-year-old white female pr...allergy / immunology, allergic rhinitis, aller...
11Consult for laparoscopic gastric bypass.2Laparoscopic Gastric Bypass Consult - 2PAST MEDICAL HISTORY:, He has difficulty climb...bariatrics, laparoscopic gastric bypass, weigh...
22Consult for laparoscopic gastric bypass.2Laparoscopic Gastric Bypass Consult - 1HISTORY OF PRESENT ILLNESS: , I have seen ABC ...bariatrics, laparoscopic gastric bypass, heart...
332-D M-Mode. Doppler.32-D Echocardiogram - 12-D M-MODE: , ,1. Left atrial enlargement wit...cardiovascular / pulmonary, 2-d m-mode, dopple...
442-D Echocardiogram32-D Echocardiogram - 21. The left ventricular cavity size and wall ...cardiovascular / pulmonary, 2-d, doppler, echo...
\n", - "
" - ], - "text/plain": [ - " Unnamed: 0 ... keywords\n", - "0 0 ... allergy / immunology, allergic rhinitis, aller...\n", - "1 1 ... bariatrics, laparoscopic gastric bypass, weigh...\n", - "2 2 ... bariatrics, laparoscopic gastric bypass, heart...\n", - "3 3 ... cardiovascular / pulmonary, 2-d m-mode, dopple...\n", - "4 4 ... cardiovascular / pulmonary, 2-d, doppler, echo...\n", - "\n", - "[5 rows x 6 columns]" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 43 - } - ] + "name": "stderr", + "output_type": "stream", + "text": [ + "\r", + "Epoch: 25%|██▌ | 1/4 [02:13<06:40, 133.41s/it]" + ] }, { - "cell_type": "code", - "metadata": { - "id": "UGJEUBCrTQJE", - "colab_type": "code", - "outputId": "3231566f-e1a5-4677-d0b0-4c421187e5d3", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 54 - } - }, - "source": [ - "description = list(df['description'])\n", - "# Tokenize with BERT tokenizer\n", - "tokenizer = BertTokenizer.from_pretrained('biobert_v1.1_pubmed', do_lower_case=True)\n", - "\n", - "# Restricting the max size of Tokens to 512(BERT doest accept any more than this)\n", - "tokenized_texts = list(map(lambda t: ['[CLS]']+tokenizer.tokenize(t)+['[SEP]'] , description))\n", - "print (\"Tokenize the first sentence:\")\n", - "print (tokenized_texts[0])\n" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Tokenize the first sentence:\n", - "['[CLS]', 'a', '23', '-', 'year', '-', 'old', 'white', 'female', 'presents', 'with', 'complaint', 'of', 'all', '##er', '##gies', '.', '[SEP]']\n" - ], - "name": "stdout" - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation Accuracy: 0.357421875\n", + "Train loss: 1.692780037720998\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "9oNK7hbPT0C6", - "colab_type": "code", - "colab": {} - }, - "source": [ - "classes = list(df['medical_specialty'])" - ], - "execution_count": 0, - "outputs": [] + "name": "stderr", + "output_type": "stream", + "text": [ + "\r", + "Epoch: 50%|█████ | 2/4 [04:31<04:29, 134.80s/it]" + ] }, { - "cell_type": "code", - "metadata": { - "id": "VPgT9lnYUJ_V", - "colab_type": "code", - "colab": {} - }, - "source": [ - "# Set the maximum sequence length. \n", - "MAX_LEN = 128\n", - "\n", - "# Pad our input tokens so that everything has a uniform length\n", - "input_ids = pad_sequences(list(map(tokenizer.convert_tokens_to_ids, tokenized_texts)),\n", - " maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")\n", - "\n" - ], - "execution_count": 0, - "outputs": [] + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation Accuracy: 0.32421875\n", + "Train loss: 1.5088001774980666\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "-cRpNSYDUWKZ", - "colab_type": "code", - "colab": {} - }, - "source": [ - "# Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary\n", - "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]\n", - "input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")" - ], - "execution_count": 0, - "outputs": [] + "name": "stderr", + "output_type": "stream", + "text": [ + "\r", + "Epoch: 75%|███████▌ | 3/4 [06:50<02:16, 136.08s/it]" + ] }, { - "cell_type": "code", - "metadata": { - "id": "aP95AoULUloD", - "colab_type": "code", - "colab": {} - }, - "source": [ - "# Create attention masks\n", - "attention_masks = []\n", - "# Create a mask of 1s for each token followed by 0s for padding\n", - "for seq in input_ids:\n", - " seq_mask = [float(i>0) for i in seq]\n", - " attention_masks.append(seq_mask)\n", - " " - ], - "execution_count": 0, - "outputs": [] + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation Accuracy: 0.322265625\n", + "Train loss: 1.4018034535519621\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "qauU8icyUngJ", - "colab_type": "code", - "colab": {} - }, - "source": [ - "batch_size = 16\n", - "\n", - "# Use train_test_split to split our data into train and validation sets for training\n", - "train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(input_ids, classes, \n", - " random_state=2020, test_size=0.1)\n", - "train_masks, validation_masks, _, _ = train_test_split(attention_masks, input_ids,\n", - " random_state=2020, test_size=0.1)\n", - " \n", - "# Convert all of our data into torch tensors, the required datatype for our model\n", - "train_inputs = torch.tensor(train_inputs)\n", - "validation_inputs = torch.tensor(validation_inputs)\n", - "train_labels = torch.tensor(train_labels)\n", - "validation_labels = torch.tensor(validation_labels)\n", - "train_masks = torch.tensor(train_masks)\n", - "validation_masks = torch.tensor(validation_masks)\n", - "\n", - "# Create an iterator of our data with torch DataLoader \n", - "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n", - "train_sampler = RandomSampler(train_data)\n", - "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n", - "validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\n", - "validation_sampler = SequentialSampler(validation_data)\n", - "validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)\n", - "\n", - "\n" - ], - "execution_count": 0, - "outputs": [] + "name": "stderr", + "output_type": "stream", + "text": [ + "Epoch: 100%|██████████| 4/4 [09:09<00:00, 137.40s/it]" + ] }, { - "cell_type": "code", - "metadata": { - "id": "ydBJogc4Uwdl", - "colab_type": "code", - "outputId": "72ad8aed-4a6b-45b2-9a0a-1d5fb4ffe53d", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - } - }, - "source": [ - "#Loading pre trained BERT\n", - "model = BertForSequenceClassification.from_pretrained(\"biobert_v1.1_pubmed\", num_labels=40)#binary classification\n", - "model.cuda()" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "BertForSequenceClassification(\n", - " (bert): BertModel(\n", - " (embeddings): BertEmbeddings(\n", - " (word_embeddings): Embedding(28996, 768, padding_idx=0)\n", - " (position_embeddings): Embedding(512, 768)\n", - " (token_type_embeddings): Embedding(2, 768)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (encoder): BertEncoder(\n", - " (layer): ModuleList(\n", - " (0): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (1): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (2): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (3): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (4): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (5): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (6): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (7): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (8): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (9): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (10): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (11): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " )\n", - " )\n", - " (pooler): BertPooler(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (activation): Tanh()\n", - " )\n", - " )\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " (classifier): Linear(in_features=768, out_features=40, bias=True)\n", - ")" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 50 - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation Accuracy: 0.259765625\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "RzGq3XXmU37D", - "colab_type": "code", - "outputId": "c54850dd-e0d4-4e35-873d-549b46609a5b", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 695 - } - }, - "source": [ - "# BERT fine-tuning parameters\n", - "param_optimizer = list(model.named_parameters())\n", - "no_decay = ['bias', 'gamma', 'beta']\n", - "optimizer_grouped_parameters = [\n", - " {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n", - " 'weight_decay_rate': 0.01},\n", - " {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n", - " 'weight_decay_rate': 0.0}\n", - "]\n", - "\n", - "optimizer = BertAdam(optimizer_grouped_parameters,\n", - " lr=2e-5,\n", - " warmup=.1)\n", - "\n", - "# Function to calculate the accuracy of our predictions vs labels\n", - "def flat_accuracy(preds, labels):\n", - " pred_flat = np.argmax(preds, axis=1).flatten()\n", - " labels_flat = labels.flatten()\n", - " return np.sum(pred_flat == labels_flat) / len(labels_flat)\n", - "\n", - "torch.cuda.empty_cache() \n", - "# Store our loss and accuracy for plotting\n", - "train_loss_set = []\n", - "# Number of training epochs \n", - "epochs = 4\n", - "\n", - "# BERT training loop\n", - "for _ in trange(epochs, desc=\"Epoch\"): \n", - " \n", - " ## TRAINING\n", - " \n", - " # Set our model to training mode\n", - " model.train() \n", - " # Tracking variables\n", - " tr_loss = 0\n", - " nb_tr_examples, nb_tr_steps = 0, 0\n", - " # Train the data for one epoch\n", - " for step, batch in enumerate(train_dataloader):\n", - " # Add batch to GPU\n", - " batch = tuple(t.to(device) for t in batch)\n", - " # Unpack the inputs from our dataloader\n", - " b_input_ids, b_input_mask, b_labels = batch\n", - " # Clear out the gradients (by default they accumulate)\n", - " optimizer.zero_grad()\n", - " # Forward pass\n", - " loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n", - " train_loss_set.append(loss.item()) \n", - " # Backward pass\n", - " loss.backward()\n", - " # Update parameters and take a step using the computed gradient\n", - " optimizer.step()\n", - " # Update tracking variables\n", - " tr_loss += loss.item()\n", - " nb_tr_examples += b_input_ids.size(0)\n", - " nb_tr_steps += 1\n", - " print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n", - " \n", - " ## VALIDATION\n", - "\n", - " # Put model in evaluation mode\n", - " model.eval()\n", - " # Tracking variables \n", - " eval_loss, eval_accuracy = 0, 0\n", - " nb_eval_steps, nb_eval_examples = 0, 0\n", - " # Evaluate data for one epoch\n", - " for batch in validation_dataloader:\n", - " # Add batch to GPU\n", - " batch = tuple(t.to(device) for t in batch)\n", - " # Unpack the inputs from our dataloader\n", - " b_input_ids, b_input_mask, b_labels = batch\n", - " # Telling the model not to compute or store gradients, saving memory and speeding up validation\n", - " with torch.no_grad():\n", - " # Forward pass, calculate logit predictions\n", - " logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask) \n", - " # Move logits and labels to CPU\n", - " logits = logits.detach().cpu().numpy()\n", - " label_ids = b_labels.to('cpu').numpy()\n", - " tmp_eval_accuracy = flat_accuracy(logits, label_ids) \n", - " eval_accuracy += tmp_eval_accuracy\n", - " nb_eval_steps += 1\n", - " print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n", - "\n", - "# plot training performance\n", - "plt.figure(figsize=(15,8))\n", - "plt.title(\"Training loss\")\n", - "plt.xlabel(\"Batch\")\n", - "plt.ylabel(\"Loss\")\n", - "plt.plot(train_loss_set)\n", - "plt.show()" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "t_total value of -1 results in schedule not being applied\n", - "Epoch: 0%| | 0/4 [00:00" - ] - }, - "metadata": { - "tags": [], - "needs_background": "light" - } - } - ] + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] }, { - "cell_type": "markdown", - "metadata": { - "id": "iXXp4l9lenRu", - "colab_type": "text" - }, - "source": [ - "The model performs very bad in terms of accuracy. Pre-processing the data and hyper parameter tuning will help us to achieve better results." + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3gAAAHwCAYAAAD0Es3SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd7QkV3Uv/u+u7r53RhMUBxASMOSHBBZgIZAxtoyBB+IZ24AxXphgG2NscHjg5Z+AZ4JIsozBYCwkgQABwkYmWShZAuXMSJpRDqPRjCZoNPnO3LmhQ53fH1Wn6lTVqdS3q9P9ftaadft2V1edDnO7du999hGlFIiIiIiIiGj0OYMeABEREREREfUGAzwiIiIiIqIxwQCPiIiIiIhoTDDAIyIiIiIiGhMM8IiIiIiIiMYEAzwiIiIiIqIxwQCPiIgWBRG5TETe3ettS47hFBHZ0uv9EhERafVBD4CIiCiNiEwbvx4CYB5Ax//9L5RSFxTdl1LqDVVsS0RENEwY4BER0dBSSi3Xl0VkI4D3KqV+Ht9OROpKqXY/x0ZERDSMWKJJREQjR5c6isj/JyLbAXxLRA4XkYtFZKeI7PUvH2vc5xoRea9/+T0icoOIfMHf9lEReUOX2z5TRK4TkQMi8nMR+XcR+V7Bx/EC/1j7ROReEXmTcdupInKfv9+tIvL3/vVH+Y9tn4jsEZHrRYSf50REBIABHhERja6nADgCwDMAvA/eZ9q3/N+fDmAWwFcz7v9yAA8COArAmQDOExHpYtvvA7gNwJEAPgngnUUGLyINAD8DcAWAJwH4awAXiMjz/U3Og1eGugLACwFc5V//YQBbAKwC8GQAHwWgihyTiIjGHwM8IiIaVS6ATyil5pVSs0qp3UqpHymlZpRSBwB8FsBvZtx/k1Lq60qpDoDzARwNL2AqvK2IPB3AywB8XCnVVErdAOCiguN/BYDlAM7w73sVgIsB/JF/ewvAcSKyUim1Vyl1h3H90QCeoZRqKaWuV0oxwCMiIgAM8IiIaHTtVErN6V9E5BAROUdENonIfgDXAThMRGop99+uLyilZvyLy0tu+1QAe4zrAGBzwfE/FcBmpZRrXLcJwDH+5bcAOBXAJhG5VkRO9q//ZwDrAVwhIhtE5LSCxyMiokWAAR4REY2qeNbqwwCeD+DlSqmVAH7Dvz6t7LIXHgdwhIgcYlz3tIL33QbgabH5c08HsBUAlFK/VEr9LrzyzZ8CuNC//oBS6sNKqWcBeBOAD4nIby/wcRAR0ZhggEdERONiBbx5d/tE5AgAn6j6gEqpTQDWAPikiEz4WbbfKXj3WwHMAPgHEWmIyCn+ff/T39c7RORQpVQLwH54JakQkf8jIs/x5wBOwVs2wrUfgoiIFhsGeERENC7+FcBSALsA3ALg8j4d9x0ATgawG8BnAPwA3np9mZRSTXgB3RvgjfksAO9SSj3gb/JOABv9ctP3+8cBgOcC+DmAaQA3AzhLKXV1zx4NERGNNOG8bCIiot4RkR8AeEApVXkGkYiIKI4ZPCIiogUQkZeJyLNFxBGR1wP4XXhz5oiIiPquPugBEBERjbinAPgxvHXwtgD4S6XUnYMdEhERLVYs0SQiIiIiIhoTLNEkIiIiIiIaEwzwiIiIiIiIxsTIzcE76qij1OrVqwc9DCIiIiIiooG4/fbbdymlVtluG7kAb/Xq1VizZs2gh0FERERERDQQIrIp7TaWaBIREREREY0JBnhERERERERjggEeERERERHRmGCAR0RERERENCYY4BEREREREY0JBnhERERERERjggEeERERERHRmGCAR0RERERENCYY4BEREREREY0JBnhERERERERjggEeERERERHRmGCAR0RERERENCYY4BEREREREY0JBnhERERERERjggEeERERERHRmGCAR0RERERENCYY4PXAfLuDfTPNQQ+DiIiIiIgWOQZ4PfBvv1iPl376SiilBj0UIiIiIiJaxBjg9cCShgNXAc2OO+ihEBERERHRIsYArweWNGoAgLkWAzwiIiIiIhocBng9oAO8+VYncv2GndNoM6tHRERERER9UlmAJyJLROQ2EVknIveKyKcs27xHRHaKyFr/33urGk+VbBm8rftm8ep/uRZnXPbAoIZFRERERESLTL3Cfc8DeLVSalpEGgBuEJHLlFK3xLb7gVLqgxWOo3JLGl6cvOPAHJ5+5CG4d9sUXD/Wu/GR3QMcGRERERERLSaVBXjKayk57f/a8P+NZZvJJXUvg/fWs2/Gn/36M3HeDY/iNS94MgCg2e5k3ZWIiIiIiKhnKp2DJyI1EVkLYAeAK5VSt1o2e4uI3CUiPxSRp1U5nqroEk0A+M7NGwEADz6xHwDQ6oxlTEtEREREREOo0gBPKdVRSr0YwLEAThKRF8Y2+RmA1UqpXwFwJYDzbfsRkfeJyBoRWbNz584qh9yVpRPh06gDupqI/zubrBARERERUX/0pYumUmofgKsBvD52/W6l1Lz/6zcA/GrK/c9VSp2olDpx1apV1Q62C5P1WuI6YYBHRERERER9VmUXzVUicph/eSmA1wJ4ILbN0cavbwJwf1XjqZJZohk332aAR0RERERE/VFlF82jAZwvIjV4geSFSqmLReR0AGuUUhcB+BsReROANoA9AN5T4Xgqo7to2jCDR0RERERE/VJlF827ALzEcv3HjcsfAfCRqsbQL0szMnhtNlkhIiIiIqI+6cscvHFnK9EU/2fbZYBHRERERET9wQCvB6xz8CR5FRERERERUZUY4PVAzUlGc4zviIiIiIio3xjgERERERERjQkGeBXR6+ARERERERH1CwM8IiIiIiKiMcEArw/aHReP7Z4Z9DCIiIiIiGjMMcDrkbUffy0++TvHWW/7p8sfwG/889V4fGq2z6MiIiIiIqLFhAFejxx2yAQOXzYR/G7OwLvuoV0AgH0zrT6PioiIiIiIFhMGeD00WQ+fTrPHStt1AQB1y3IKREREREREvcIAr4cm6vans9VRAOzr5REREREREfUKA7wemqzXrNfPNDsAgGbHheuqfg6JiIiIiIgWEQZ4PWRm8DpGIDfbbAMAXv+v1+Ofr3iw7+MiIiIiIqLFgQFeD5lz8NpGgNfsuMHlH92+pa9jIiIiIiKixYMBXg+ZGbx2JwzwWh2WZRIRERERUfUY4PWQOQdPd84kIiIiIiLqFwZ4PZQ2B8/EXB4REREREVWFAV4PmXPwWJZJRERERET9xgCvh4pk8IiIiIiIiKrCAK+Hohk8+xw8xbiPiIiIiIgqwgCvhyZq+QEeERERERFRVRjg9ZCIBJfTKzSZwiMiIiIiomowwCMiIiIiIhoTDPCIiIiIiIjGBAO8PmOTFSIiIiIiqgoDvCFz/+P7cdndjw96GERERERENILqgx4ARb3hy9cDADae8cYBj4SIiIiIiEYNM3g99qk3HT/oIRARERER0SLFAK/H3v1rq3HC0w5LvZ1T8IiIiIiIqCoM8CpQk/TbFLusEBERERFRRRjgVcCRjAiPiIiIiIioIgzwKuA4DPCIiIiIiKj/GOBVoBfxneuylJOIiIiIiMphgFeBWkaEVzRsazPAIyIiIiKikhjgVSBrDl7RHitt1+3RaIiIiIiIaLFggFeBMk1W5tsdzDTbieuZwSMiIiIiorIY4FUgq0Qz7o1fuQHHffx/Ete3OwzwiIiIiIioHAZ4FSjTZGX9jmnr9SzRJCIiIiKishjgVSB7Dl6xzBwzeEREREREVBYDvAr0YqHzDufgERERERFRSQzwKlBmDp7JzO61OizRJCIiIiKichjgVSArgZeVl2saQR0zeEREREREVBYDvAp0m8FrtsMAj8skEBERERFRWQzwKtDtHLxIgMcmK0REREREVBIDvApkBngZcZtZosllEoiIiIiIqCwGeBXoskKTJZpERERERLQgDPAqkDUHLytsm2eJJhERERERLQADvAqIX6JZL5nKi2bwWKJJRERERETlMMCrQM1/Vhu15NM7Pd/Gxl0Hrfebt5RoKqW4ZAIRERERERXCAK8CNZ3Bq9kzeKd84Rrr9bYumh+6cB2e/dFLeztAIiIiIiIaSwzwKqBLNG0ZvCzRhc69yz+5c2vvBkZERERERGONAV4FdJOVsnPwZpvt4DK7aBIRERERUVkM8Cqg47qsbpo2M81OcJldNImIiIiIqCwGeBVw/MAuc8Fzi0iAxwweERERERGVxACvAjqwKxnfYcYs0exwmQQiIiIiIiqHAV4FdBfNrAze1EwrcV1WBk8pZvSIiIiIiCgbA7wK6Kl3WVPwTjj9isR6eLOROXjRDB7XwiMiIiIiojwM8CpQdA7eY3tmgstKKRxsttHw186LZ/A6zOAREREREVEOBngV0IFdXkg2UQ+ffqW8Es0VSxoAbCWaPR0iERERERGNIQZ4FdDLI+TNm9PZOgBwlcJss4OVS+oAkiWZLNEkIiIiIqI8DPAqULR7pjnNzlXAwWYHK5f6GbzYOnguU3hERERERJSDAV4F6n4GL2/e3FwrbKriZfDaWDZRhwjQdqNNVlyumkBERERERDkqC/BEZImI3CYi60TkXhH5lGWbSRH5gYisF5FbRWR1VePpp+WTXhbO7IppM2sEeHoO3iETNTQcJzEHjxk8IiIiIiLKU2UGbx7Aq5VSJwB4MYDXi8grYtv8GYC9SqnnAPgSgH+qcDx9s3KpN4/uwFw7c7t4Bm+m2cEhk3XUHEkuk8AAj4iIiIiIclQW4CnPtP9rw/8Xj1J+F8D5/uUfAvhtkaIz2IbXSr8T5nw7u67SDPAUgJlmG4c0aqjXpJIM3l9dcDtO+NQVC94PERERERENp0rn4IlITUTWAtgB4Eql1K2xTY4BsBkAlFJtAFMAjqxyTP2gG6XkMUs4dQZv6UQNjZqDVqf3c/AuvXs7pmZbC98RERERERENpUoDPKVURyn1YgDHAjhJRF7YzX5E5H0iskZE1uzcubO3g6yAXuogzyd/dl9wWblAq+Niou5gouag1eYcPCIiIiIiKqcvXTSVUvsAXA3g9bGbtgJ4GgCISB3AoQB2W+5/rlLqRKXUiatWrap6uAtWNINncpWC63qLpDfqgmZ8Dh7XwSMiIiIiohxVdtFcJSKH+ZeXAngtgAdim10E4N3+5bcCuErlrQ4+AlYUzOCZXKXQUQo1B5ioOWjG5u+N/rNCRERERERVKx+JFHc0gPNFpAYvkLxQKXWxiJwOYI1S6iIA5wH4roisB7AHwNsrHE/fTNZrpe/jKi9LVxPBRL2WzOAxwiMiIiIiohyVBXhKqbsAvMRy/ceNy3MA/qCqMYwSXYLpOIKJmiQyeJyDR0REREREefoyB4+ivv6uExPX6a6ZXgYvWaLpcg4eERERERHlYIBXkRWTdTx71TLrbRP15NPeNjJ41mUSGN8REREREVGOKufgLWprP/E6CIBnffTSxG0NJ7mWe1tn8Bwvgzc9347czi6aRERERESUhwFeRWp+EPfXr34OBMBXrlof3GZbRqHV8QK4uiPWLpq9nIOnlIJIMsgkIiIiIqLRxgCvYh9+3fMBhAHeWe94KV54zKGJ7dquF9B56+A5iS6avQzwXAXUGN8REREREY0dzsHrs1NfdLT1ep3BqzmCyZqDbftmcfeWqeD2XlZo6mCSiIiIiIjGCwO8IaHn4Dn+HLy5lovf+eoNwe29nIPH+XxEREREROOJAd6Q0F00a+J10YxTPSzRZIBHRERERDSeGOANiWAdPMe+jAIzeERERERElIcB3pBo+3PwHH+h87jezsFjgEdERERENI4Y4A2JoETTsZdo9rKLJjN4RERERETjiQHekNCdLWuOYNKawWOAR0RERERE2RjgDYlIiaYlg8c5eERERERElIcB3oB9410nAjCbrAgallXIe5jA4xw8IiIiIqIxVR/0ABaLdZ94nXWpA8cPsXXQ5TVZqSW2620GjwudExERERGNIwZ4fXLo0kbk9z866el4ZMc0RLxsXdvI4Ckkg7nezsHr2a6IiIiIiGiIMMAbkM+/+UUAgGsf2gkAaHV0F02g1U5GYL0M8NrM4BERERERjSXOwRswx59up4MuR8Q6R66X0+byyj13HJjr3cGIiIiIiKhvGOANmOOXaIYZPMG8JYPXyzl4WU1W1m3eh5M++wv815rNPTseERERERH1BwO8AROdwdMBnkjQUdMUL9G8e8sUPvWze62NW/K4GQHeIzunAQA3rt9Ver9ERERERDRYDPAGTOA3WdElmo7gna94Bo5/6srIdvEA723n3Ixv3bgRM81O6WNmZfAm/EXWdUaRiIiIiIhGBwO8AdNz8HRAVXcERy6fxIV/cXJku3hflM4Cmq5klXs2/EXWbWWiREREREQ03BjgDZjjRJdJ0L8vaUTXwosHdLo0s5tFy4tk8JpcS4GIiIiIaOQwwBuwsItmOAcP8JqtNGoSbBefa6ezcFnz6dJk3WfSz+A12+VLP4mIiIiIaLAY4A2YBF00w4XOtSX1MIsXT6jpGK2bUs1CGTyWaBIRERERjRwGeAOml0nQXTT17wAwaZRp6iYr//rzh7D6tEuC67tZPqGTsdC5xOYEEhERERHR6GCAN2A6YbfeX57AzOCZJZo6wDvrmkci9+8mwMvK4OmbBpXB++7NG7H6tEuYQSQiIiIi6gIDvAHTGbvbN+0FANSMV6RuBnh+5DVZj75k3WXw0u+jKz4H1WTlX658CABwcL49kOMTEREREY0yBngDZlRkAoiWaDac8OXRFZOJ7po9D/C82waVQVvA6g9ERERERIseA7wB0wuda2aJZt3SRTOewYuXWz4+NYsv/M+Dia6bWfcx6VsGvUxCPPAlIiIiIqJ8DPAGzIm9AmYGr25m8FJKNN1YIPf+792Br169Hg89MZ16zKwMnjvgDB4REREREXWPAd6AOZKewYs2WfF+xks027Ful7un5wEAS2PbmTLLOgfcZCUr80hERERERNkY4A2YEytFjJZohi+PzqzFA7x4Bm/ab04SzwyaMufg+T/njYXOz7n2kb53toyXri52N67fhc17ZgY9DCIiIiIacgzwBkxiGbxoiWayi+aSRvYcPN19UingoScOYO3mfYljZs7B828yN/nq1esBALPNjuUe1VBgJs/0jm/ciledefWgh0FEREREQ64+6AEsdtklmmYXTT0HL7uLpl6g3FUKr/vSdQCAjWe8MVL6mLXQeTwjCCBM6/UxqdZFc1AiIiIiokWPGbwBS5RoGgGfGeyFc/CKrYMXv9r8PatB5qDjKn18zsUjIiIiIiqPAd6AxTN45tw5M4MXLnRebB28eCbO3C4rgzcsgdVwjIKIiIiIaLQwwBuw+Hpv6V00VeJ2ALh90x584II7EoFePFAzA74i6+DZruvn2nTWUlEiIiIiIsrEOXgDFm+yktZFU8/Biwc+X7jiIQDAh173vEiXxblWNEunIiWa+cskWG/qZ8zF+I6IiIiIqDQGeAOWNQevYdyog6u0IOvDF66LdMzcfbAZub1oBs+WOdPZwL6Ub6rIDyIiIiIiKoElmgOW1UWzbpRo6qxbWvbtkR3Tkd/1gufB/Y3gLKv80bxJr3unLLdVjRWaRERERETlMcAbsPi8NrNk0yzRbLZdTM20Cs9N2z0dzeAps2KzYIXm3plm6m1V4xy80CAb33zsJ3dj9WmXDOz4RERERFQOA7wBi2fwTGaJ5ndv2YQTTr8iNfCJX7/r4Hzq7dkZvPC2eJDYj6BLxX5SzpzJil1w62MDOzYRERERlccAb8CyAjwzg6elnezH59VNzbT8/Xu/m8FZVpxm7maPP48vb/5fFYZluYZhwEXfiYiIiKgoBngDFm+yYqpbbkw72Y8HftPzbQDAkoa3bl50Dl7WiIwMnp8FVP51aoF5tT0Hm5hrdQpty/guxHJVIiIiIiqKAd6AxZdJMDUsGTy34MLmB2MBnnlzVqBmbrc31olzoXWTL/30lfjDc28ptG1WTPP/fno33v3N2xY2mBHCAI+IiIiIiuIyCQOWmcGrJW9MW+IgfvXBeS9TprtyFi3RNG/ak1hqIf1+Ra0zlnKwHl/lZwu/d8vimhc2yDl4mlIq88sIIiIiIhoOzOANWGaTFUsGr+26li2TDvgZPB0wmTFC1vw2MxDcHZ+D18fWJ0xahYYgvhuKMRARERFRPgZ4A5aVFFk2UUtc1+4ovPTph+Fr73hp5n51iaY+MTdLO7PO1c3Aamq2lbit1XHx4QvX4aEnDmQePy6ttDR1e0Z4gbLPXRWGIYtIRERERPkY4A1YVgbvj17+dHzwt56D5zxpeXBd21WoOYJXPOvIzP2GAZ6K/IxfjtO3iJiZu/B+dz62Dz+6Ywv+30/vyTx+XLNTLPMYHwdFG+QMCgNuIiIiotHAAG/AsjJ4k/Ua/v5/Px9HLZ8Irmt3XDgicLIm7yHsoqmzP9ESzfT76fJNRyQsyTSWSdh5wOusedjSRubx4+ZbxQK8YB08xhOBYQiuhmAIRERERFQAA7wBy8rgacsmwl44rY6CI2JdQsE03/YCKn1iHs3gJbffPjWHEz9zJR7ZeRAAUBOBbbrfrmkvwDtqxWTuuKPjKbY8QogRhVZw2mW1Y2CER0RERDQSGOANmBngPfiZ11u3WWrMxWu7LmqOBN0x8+gT82hjleTJ+tZ9M9g13cTGXV6A5zjefTfvmQnKK12l8PjUHABg5ZKSGbx2uSiFU75CwxBcDcMYiIiIiCgfA7wBM+O0yXqyqQoQzeC1XQURlAjwvJ/mFDhbRijczrtQE4EC8Kozrw62UQrYsncGANDMCdguXLMZn774vuD3ohk8paI/aTganAxDFpGIiIiI8jHAG7Aia4sdMmlk8Dpek5VawTXJOpYmK7blDvRcPb0Mg+NIonujArBjv1eimRew/cMP78J5Nzwa/D5XcA5e1hiB7CUextUwPGRm8IiIiIhGAwO8EXCIUaLZcVXQZKVIjKesXTST2+nr2h0/g+cIdsUWOldKoeUHgGVLLvX2hTOPKbsve9xxwC6aRERERFQUA7wRsLQRBngtv4smgEJZPNdS8mg7V9eBYNso0dyyZyaxL33fuVa5pik641c0wEvL4BXtxvnfa7finq1TxQY35IYhuBqCKlEiIiIiKqCevwlV7f++5nk45fmrUm+vOWEc3nZVMG+v5kgQkKXRwUHHzS7R1NeYJZp6SQRzKx0IdpvBy+v+qceWFtPMFgws//Y/1wIANp7xxoIjHF7DsND5YiyNJSIiIhpFDPCGwN++5rmZt5tBUbvjBlmwvGzY0kYNs60OlFLROXjWEk0/g9cJM3hxSoWZnNIBXqtYgJfHzBwqpQrNYRx1w1CiOQxjICIiIqJ8LNEcAeai5m1XBb/nBXiNmne7GZh5v1uarOg5eK5e6Dy5P1eFgWDZEk291EKjVuwtl1aWOGc0dxmCxFZfDEMHy8XyXBMRERGNOgZ4I8AMttr+QudAkQDPe3ldpSJBnb3JSnQOnmPZt4JaQAbPC8zqtYJz8FICCrMb5zAsH9APQzEHb5E810RERESjjgHeCDADuZbrQsdIeeWOOphyVXwOXpIK5uqld7tUKtxuvnSTFV2imf2WC9bBS7ndzBwu1gDvtkf34ItXPNjXMQxBjElEREREBVQW4InI00TkahG5T0TuFZG/tWxziohMicha/9/HqxrPKHvZ6iOCy0ohyOA5OfPPdDDlKhXJ2tkyQroMMH8O3sKarDQKZ/BSSjSNAK89DLWLfRAPZN92zs34ylXr+zqGYcgiEhEREVG+KjN4bQAfVkodB+AVAD4gIsdZtrteKfVi/9/pFY5nZL3g6JX44ttOCH7vZg5eJGDqskTTDBTLZ/B0iWbROXj2680SzSLx3TiUFg7DQ2CTFSIiIqLRUFmAp5R6XCl1h3/5AID7ARxT1fHGnRkYmcskFLlPoQxesNC5X6KZkh3sOoNXsItmdiFpGCgCxTJ40812keENtWHInnGZBCIiIqLR0Jc5eCKyGsBLANxquflkEVknIpeJyPEp93+fiKwRkTU7d+6scKTDy4yLii6ToIMpV6lIBkZf3LxnBrc9use/rkCTlQUtdO4GY4nbeWAeU7OtxLFsdAkpUGwO3v7YfrOs3bwPf3jOzZEgchgMw1zDIRgCERERERVQeYAnIssB/AjA3yml9sduvgPAM5RSJwD4NwA/te1DKXWuUupEpdSJq1alLwg+zsz5dlK6i2Y0sFJQ2HuwiVedeTXeds7NUCpc+lwHE7ZKStdYT69oBm/DzmncuH5XEDTZgpW/+O4afP7S+2PHsu/PDFSLlA3uny2ewfvIj+/GrY/uwfod04Xv0w/DkMEbhjEQERERUb5KAzwRacAL7i5QSv04frtSar9Satq/fCmAhogcVeWYRlUkg6cDvLwmK8EcvOQyCXdtnQp+f2L/fHAC38pqsoLoXD1dzpnl1f9yLd7xjVvRDDJ4yW32zbaw52AzeqyUgMKcU2dm89LuG88MZtH7zmte02/D0EtmGMZARERERPmq7KIpAM4DcL9S6osp2zzF3w4icpI/nt1VjWmUiRF0FC3RbPhdNDuuipygKxUNlNbvmDYWOncTxwvvF91PmXl4ev+2DJ5SQEsHiznLJEQyeClpPjM2bBUIQuP7znte+y0te9bPeXHM4BERERGNhnqF+34lgHcCuFtE1vrXfRTA0wFAKXU2gLcC+EsRaQOYBfB2xW4OVrVIiab3M2/R8Mg6eP7TKuIFBu1IgHcAhy+bAAB0OulBjhvrxjnX6mDZZLG3ULjOnq3BS3Q83vb2/ZibpZVodiLZyuJvJ73tsGXwzNfO5Cqg4KoTC8YAj4iIiGg0VBbgKaVuAJB5+qmU+iqAr1Y1hnFirg+ug71DGtkvn+6iaZZo1kSgEA20Nu6ewaGHNACETVbs5Z/RbpzxoCyLDhD0z8d2z+CebVM49UVHw1UqKOEMjlSgRDMtg+d2G+AFJZqF79IX+rmID8tVCrXs/2I9wyYrRERERKOhL100aeHMkknd4XL5kuwAr+GEGTx9gl5zJNIsBQD2z7WC0ksdNDmWd4ZSXoMWHQCVKX/UwaDe/2u+dC3+6oI7vPG5yWBRwbmF9yUAACAASURBVAtsPn/p/di0+2BwfafQHLzwcpm5Y+ZzNEz00xzPLPYzq8YMHhEREdFoYIA3IsyTe315eU55ZFiiqYKgp+YIlDICOQEOzrfDJit+RGQLcrwmK8BE3XvbpAVYNnodPH2ceMZOB4u6n6dS3tzAc67bgL/47u3BdkWyc91m8DpD2mRFjys+rF7GXPdt24//uXd74np9TFZOExEREY0GBngjwoy39OW8DF50ofOw9NL8feXSBmaanSBY0D9tQY7reqWek/UagHIZPL1uXtocvFYsWHQtSzfEL8ezfg89cQA/v++JyDbdzMEbNmGJpmDjroPG9b07xqlfuT4SSMexRJOIiIhoNDDAGxGOpYvmsola5n10iaYy1sHT5Z06CFq5pIFpI4MXP4YpnsGLB2VZZnMDvNgcPONyx7g90mQlVn/5ui9dh/d+Z01kmzKBiX4Ohi3OC5rGCHDKF64Jru9HQKrfBcOw2DoRERER5WOANyLMhJqej7e0kR3gmRk8Tc/B09mvFUvqmJnvJJYlsK6D5weKE/5+2yUmuOkALx4nKOU1btFr6umhvvubt+H7tz4GANiw8yBecvqViceSlkCMrvlXpkTTv3/qIg2D4YbxXez6PgR4Epb5EhEREdHwY4A3IiIZPP/yEksG72WrDw8uN2pmk5Vwfpm5Dt7KJQ0cbCYzeI4tg6cUXFdhsosM3pw/By+eCdJLL9j29cPbtwSXp+fbWLd5X6xE0x7hRbN8xceohjSDl7YAez+SavqIw/acEBEREZEdA7wRYZZM+gm0RAbvvHefiO/+2cuD3+uOMQfPDe+rVFj2t2JJ3W+yEj2erZGk19nSbLJSPIM3r0s0VTzA8zJ4tvl88cYev9y4J3U+nsm8vkxgosc2bLGMDr6TTVZ6P9K015QZPCIiIqLRwABvRDiWEs1DYhm8miORLI/uoqmMhiVBkxWdwVvawMFmJxEsWOfg6RJNP8CbbSXvlyatyUrHb9xiDfBiv8eXd0gL8Lov0dQZvOEKZoIumrHrq8jgzbfzs6JERERENLwY4I0IsTRZWRLL4NUdJxIINoI5eGGgU6tFFzpfsaSOZttNLFtg7aLpZ9v0HLz3fOuXOOuaRwqNf65tL9H0gsZwyQUVu83UceNz8NKWSUDuNjZBJ9HC9+iPMIOXvw6eUgrnXvcIdk/Pd3Ws+PtAH9JlhEdEREQ0EhjgjYjoOnjez3iJpuNEt6s7YYMMnZWqiUApBT3lbeWSBgDgwFw7sq/0LpoKk43wbXPhms2Fxj/b7ASXzWBBZ+Valvl08QDGVSrSWCW+TILtfqVKNIMMXvH79EOZJit3bZnC5y59AB/+r3VdHSuewROwyQoRERHRKGGANyKi6+DpEs3oOnh1x4nM09JdNL0ySP++TqzJytKUAM/aRdPbj87gmWPJM9cOAzzdURMIF2G3NVlJBHhutEQzLau08HXwhiuY6aREeLaHpktd469nUfPG62Qekwk8IiIiotHAAG9ERDN43uWXP+sIvPFFRwfX1xyJlPFF18Hzt5HoMgkr/cXSp+db0eNlZPD0HDwgLOG7+K5t+O4tm1LHbwYj++fCY7muzsypRMAWz9B1/O3SbrcdaxzWwQvmBCaWmLBsm7Gfh584gD869xbMNNODv3iJpsYMHhEREdFoYIA3IsxEmS6fbNQcfO7NL0pcr5nr4AVz8BwJAjUAWOGXaE7PR0/6rV00/f1M1GvGdt6GH/z+nfjHn95T6LGY2SWdwQOQKNOMxxSuqwp10YzM0yuVwfOPW/ge/aEfp60DaRpbXvWzl96Pmzfsxi0bdqfeL1mi6Rm2xjNEREREZMcAb0TY5uDFL9djUZm5Dp4Kthe4KtpkBQCm58PSPBF76aXOBOr9xo9f1AEzg2cEn+2Oygwk9Jp5WsdV+Pp1G3Dftv2R7cxAqExgUnYO3qbdByPZyKLufGwv7nhsLx7bPYMNO6dzt9eBp21OYhrbLXrO5mwzfXmLRIDnv74lVsQgIiIiogGq529Cw8DMzpnlk46lu6ZmbbLiCGCUOjaCeXrhGbwgueaatx/v52Q9Ogdv78FmqccyNRsGRR0zg5cTRXSUigRvHVfhs5feDwDYeMYbg+sjyyR0MXlMFczh/eY/X4Nnr1qGX3z4lFL7//2zbor8bo49MRalcN3DOwEA8T40ZZNquuuqOQcyLj4Hj01WiIiIiEYLM3gjwtZkJX45rURTqWiTFb1sgki4aHrbaHIiIomW/ECY4Zo0SjRFBBt2HSz1WLZPhS389dp6gNdoJSuM8Eo0w9/Tu2gaY+4iLikTyzyys9xjL+sHv9yMax70A7wFlmjqAG8uM8DjHLzF4IaHd3WVfSYiIqLhxwBvRETWwTMDPOMVjAd4ZolmMAdPvAxVx1WoGYGcGSwJ7KWXeh8TkQwesGXvTKnHsn3/XGSfYYCXncHTC7TrzGTbsrQCEJ2bt31qFq868yo8WiIIHaZg5vGp6HNlKpucXFogwEtbB2+InhJaoF3T8/jj827FX3//zkEPhYiIiCrAAG9ERLJ2KSWa8Tl4dT/685YX8K6r+cskdJRCzZEgWDQDPEckKM0z6cApvkzCfKvcBK3tU7PB5ZM/f1UwtnZOuq3jeuPWZaVpHR/N/fzojq3YvGcW59+0sfCi58MUzJiL2ceHXyQQ/eYNj+Lsa73F6JdOeM+buSZhHDN440+//ut35M//HFf3bJ3C6tMuKfXFDxER0ahggDci0hqrpJVrAkDdbLLin6AHTVY6XoCn72POwYOUy+DZFinPsn3/vPX6ZpEMnrFMg23tPCCa2dvjzw/89k0b8dazb7Ju341+dZVc0kj/L2obQ/yq0y++D2dc9gAAI4MXX+vOMN+Kz8HzFA2Ox9m2fbPYOOYBwZqNe7B72v7/c5x85RcPA/AaHhEREY0bBngjIm2uXaSLZi1eopmcg1evCZTfrMQr0fSuj8zBA6xz8PQ2jZq5Dp5E7qtUdidMAHjCKDuM7N91M7NnukRTH/+fLn/Aul1aMHLnY/syx6UVid36Fe+YGbwyY7A1yVlSoItmPMjW74NRT+Dllf8W8WtnXIVTvnDNwgczpGaabbz17Jvxge/fMeihVO62jXsAAE9asWTAIyEiIuo9BngjwjxhN4MvKdRFM8y+OeKVaLqugmNk8CJz8MQeIOgOlkZ8B5HofTuuyg0GzDl4plY7r0RToaOiXTxt0pqvFFWki2YvAoYisjJ4mcskWG7Sz1tmF82WfR28US7RvHXDbjz3Y5fh1oz1/wi4//EDAIBd0+W64o6C7VNzeNc3bws6+O6b8X6O8vuaiIgoDQO8EeGkNFkxxa9vGAud69MYb6FzFc7Bc3SJZnQOnm0dPL3kQLwstG0EO82Om7u4+NRsKzKPT8sr9XT9wLRRS47NbBySN5cvT5FzvoWULC6bSM/KxdWcrBLNcsfVm+vnqtl24brRjGv6HLxyxxomNz7iBXY3PcIAL8t926YAAC84euWAR9J7X736YVz30E5ctHZr5PoRflsTERGlYoA3IqIlmvZt6rFgIJyDp4wumnqhcx3IeduaAUtaF02dGYt39DQzZq22SnwrbtvXUkuQkxeYecskeHPw4kGSubZe2TmBcUVO+haSJTx0aaPwtlnr+GUuk2B5zvXmusnG8/7fZTjtx3dFXvtE4xrJPxaNh/v8DN7Rh45f2aL++xn/79SvubRERET9xABvRJixWyMlwosne2qRhc71Nt4cPNdVqDlhsGZm4dLWwQvLPI0rJRqYNTtuYkFuc908bcJSZllkoXNXKTgi+PLbXxK5TZdcAV4DmYUoctK3kAyeY4t4uzhO1jCbbRe/f9aNse29O8y2OsHlC9dsiQSryYXOPQzwxt9+/0uScQx6wgAv+tjG8KESERExwBsVZgYvLcCLZ/B0GaTrhidtNQmXSag7ThCsxdfBs87Bs5Roelm1MDBrddzESZQtmLPNoyu0Dl6stFQzA5Mi2bWXffbneNs5N1tvK5TBW8AcvJQKW6usctesoOuRnQcjTWVcNyzTnW11IoHjjLFswlx8Dp4+MR7hGk39dI/uI+gP/Q4Z4Zc6VyKDx3cFERGNIQZ4IyKy3p1lDhpgabJizMGLrIMHL1hzHFjn4HnLJKSvg2cepuUqtGIlfvGgRAd45vjsAV6xEs2aI4ksmBnUpS2Abtp5YB63PbrHeluRb/UXUqJpW2MwTVY2xTYEvX38CPNtN9h+vtWJjP83z7w6uLxvxt5gY5xP+smj/9uMY7ZW/z2L/38aw4dKI2bt5n1jmTUnosFigDcizHjG1qAEKNhF0y/R7LjeMgm2LpreQudJQRBoBH/tjhvJZrU6LlQsvtLjNZujTFjKNvOyYh3l/RORREMZs0x04Wu2VVuiWSqDl/GU2E7Eg2tix5gzyjLjGbwD8+3g8hMHoh1O9VjH8aR/sYu/D/VrPMwvdbvjYlcX6/Q5Ke9jfnFBg3TNgzvwe/9+I753y6ZBD4WIxgwDvBEhBUo041O7dKbPXAevJt5JTUd5yySE6+CZc/DsGTx9MhTJ4HXcSHBo66Kpx9EwSkhtGbxCC527CjXLQuxm1q4fXTQXksGzPbdpsko0iyx0rpmLm0/PtVP3+0TKIvTDfNKfp0xAvZjpl3iYg/mP/eQenPiZn0e65hbhGF92mZg5oUF6dNdBAF5JPRFRLzHAGxFZC5pr8cYoZuc4pVQQuCnoQCllHTwkAyhvP8k5eO2OigRUrU6yi6bOtpnjtgV4RbtoWks0e5jBq3wOXolts7toJq/Tc4rix5htdoLXZceBeWsjmuWTdTwRW6NQ7ydv6QsafTrYGeYA7+K7tgEo/wVLapOV3gyLqCv6cytefUNEtFAM8EaEGVSllWgm7+P91HPwBADEm2ujAyXbHDxJWelcfxhF5+BFM3itjpsISnQwZmYeu+qi6YZdNOMfiOb4F7xMQokMXjcfzGVOKrOCVVvwZ3ZLNc213OC2tqswbZRlasccthQH5tqYaYa3ScqJ8Ugah8fQA2lPg347DXPZYlAlXvJ++s9Z/LEzg0eDpD9H6gzwiKjHGOCNiGiTlWIvm3lyrqAiC5jrQCko0YzMwbNn8HQWx8wUttoqutC50cxD0xk8M8CzLZ3Qyjmz1HMJa44kxmcGhwsv0Sw+B6+bz+UywZLe9oRjD03cZttL2r5nW53I9pv3ziS2OebwpQCAHUaZZtCBcoTPg8s0temn/XOtgQYY8e9w9FiGOegJ5gmWvJ8TBHhsskLDQ3egZgaPiHqNAd6IiK6DV+zDIOwc5wVHunmKqxTarkK9JinzwezXu0FQY5Rouvlz8HQ2yczaWbtoxhfZjh/fbw5jBqqamenKK98yA9I7H9uLe7ZORW4vcs6nA8oy8+m0MgGeflzHH5MM8Gz7SXvo861OZPste2YT2zzFX+Da1sSi6DIJt27YjV/7/C9w0JIhHLRhOpd/fGoWv/LJK3DudRv6fuy0pQGCDN7CEuCV0m/hskFovFw92F/PRkZUXpDBK/ilLRFRUfyrMiKKrIOXvI/3U68fp5c/UApBoBTvRgnoJivJ/bWNrNUpz18FwJtzFynRbCdLNPVwJ3MCvLzlDTquQkd5AWP8G8/IMgk5pZ5mM5ffP+sm/J9/uyFye5FzR9uagGlaHTeSYSxzbqqDZVsJj20/aZ0Q59qdyHW2DN6yCS+r2ow13PH2W2y8Z1z+ALZNzeGB7QeK3aEPhrHJyrZ93lzHy+/d3vdjp73/RqHJiv7/ULaMNFLNYNx3mB8rjT9dbcISTSLqNQZ4I6K7AM/oHKe8wEwkumC4LUDxFjq3ddEMg5pv/8lJeO+vP9ProhlZJiG9ycrSibAsc7LRxTp4kS6a8QDPKNHMOftr5mQKiyx+XGYO3q9++kq89ovXhvsvcU7pZhzHukyCPgGOPQezzehjfnwq2kwFCL9Fts37K9pkRd+XJyzZnJKBcy+lHTIo0ezfUEoL3vMlBxk832707xPjOxqkMIPHv5dE1FsM8EaEeb5ctERTYhk8gUAkXOi8JgKxvANSeqwEJ+/6tnrN8bpougpL/ICt1XETAYIu0VxizLuzNYrJa7KSFZiW6aI5nxPgFTl5LDMHb/9cGxt3hxmzciWa3k9bUG97mDrOjQdkc61OJOiztZnXga8ZIJctiRvmrnDDdDIffPkygAgv7f3nBtmxIXqiYvTQyo7R/LLLfMq7eaTXPLgDz/vYZTgw1+ri3kQh/eUovxCjqjTb7lBOmaDqFQrwRGSZiBcKiMjzRORNItKodmhkKrIOXlw4B88rS9IZPKUUXNeb12fL4NnmuAHAeTc8GtlvoyZeF82Oi0Mm6gCym6yYGTxbF8285igdN5yDl1miueAMXj4djJYJZHRQWCrAsyxNoWUtdB6/ba4dbbJiC3L182IuoVD2pL/Db6QLSWvb3w+pJZoq+/ZhUj7A834qxDN45R/sl658CM2Oy7XLaMHCLpr8rp2q8Qfn3IzjP/E/gx4GDUDRvyrXAVgiIscAuALAOwF8u6pBUbZuSjTNJitKeYFDzUmZg4fsNuRhgOdAKS9YWNoI52/FT76CDF4jp4tmbgYv7P4ZfwqKzsHruArz7exFksvMwUsL8K64dzvefu7NkRPIbftmC+8/HItK7WpqOzkNA7Lo9eYyCXVHrEHuk1dOAohl8IL9FhuvLpW1va8oVHZuY2+NbgZPKztCMf4Wmg+vm4fK1vbUK3mfI0QLtW7zvkEPgQakaIAnSqkZAG8GcJZS6g8AHF/dsChL0Q8DvZlePy5osgK9Dp5jLcWUlAxefL86SzPX6uAQPzvXsgR4+mR/ScOYg2ddBy9nDp6x0Hl8jqAZ1GVl8FodF3OtZHAT7axXfA6eba4iANyzbT9u2bAnMpb/+4O1OPvaR6wn9f+9diu2W+bF6Yyl7TC2njRpc/DMuUf1WjLAe/NLj8Frj3tKcEzzfkD5DN4wxXfBUg9DNLvMzK732yivg6d1X6IZy+B18Z4o02CJKMtC1lMlIspSL7idiMjJAN4B4M/865IpGBoqOnOmTyJ1oKCXG7A1K9Gyzl10UNPwy0pmmtEALz4HTn94LW1kl2jmLnTuL9juWDKPkWUSMgLFtqsiXSJt9y9yght88xobx84D83hk53QQGJljWbNpL9Zs2osjl01E7jPX6uBv/3MtnnXUMlz196dEj6MUHEesa7llLZMQv81bC9FTd5zEc/Brzz4qeJ3MhjVBBq/gWX87CAgLbb5o6YqsQWTLUl+bLue3DULZIZoLnZuPr5slIViGTL2iv5gsOq+eiKioohm8vwPwEQA/UUrdKyLPAnB1dcOiXoiWaOpSv3CZBNuC4YB38pmWmfL26/1sGBk8nZ1rtt3EyVfRdfDylkkIFjqX5DeeZqask7GfdsfFvCWDF20skn/2GK6DF73+zV+7EW8/95bgJLJlGUt87/pwW/cl16Zz/WY4ttfJ2mQlpUSzo7zaNPGfu3gGr+aEJWedyHORfiybbuYZjgK35Psjj/5iIK8hUBV0qB//0iBcYmP4X7tuh6j8L4mC37vYR5UZvN3T8/jxHVt6vl8aTmGJJufgEVFvFfqropS6Vin1JqXUP/nNVnYppf6m4rHRAsXXwZNYmZKtWQngnfhlVYzoExvdVn+u5QYNVJodlczgif4Z7nSykUwAN9vFSjRtJYuRpRoySzTtc/CiZVv5gpM8RzDX6uCzl9yH6fk2NvsLiOsh2BZvjwc/WYFtx/WDWevJZHoGL3G9653YCtICPMfI4CWDmaIBW5DBG6LFss3sTbfM52T7/jls2t2bBhvmmB6fmsWZlz9QeWfN3HXwhui1S1P2CwS9eceNlWN38aYI3wu9f53e/73b8aEL1+HxqeSXPdR75173CN71zdsGdvxWEOANbAhENKYKlWiKyPcBvB9AB8AvAawUkS8rpf65ysHRwuiA7lM/uw8AcNTyCS84MzJ4tkydZJRu6tsB4LBDvEaq2/fP4UXHHoqJmoNHdx3Evdv2R7bX+3KMqNG2TEKRhc5dV+UudN7JKNFsdVxrg5F2ya/1zbkT37/1MXz9+kcjzW90FtHWrTJ+Ap819zDMvNpuS16XdsLqKi9zI35QHy/RrIkEJWe2ctWi58HjmsEzn5OTP38VAGDjGW/sen+2Utq/+8+1uPXRPXjtcU/GS55+eNf7zpM+B290XruyIzQf20KbrITv8fL3zbN9vzcPN6/TL/XG5y59YKDH158TI/BfjohGTNHvjY5TSu0H8HsALgPwTHidNGmIxYMCMefg+XO7rNshew6eDtie86TlwXV1R9CoCX62bhs++pO7o9v7BzD32c0cPD3umiTn4Jlz3bKarLQ7yhp0mUFhoSYrnbBMa87PCJprz+mgzbb+TPzDXGcfbUcNAvGCc/DSThQ6brhURk1sGTyxZvDKnvQHj2XMTliKLvRelH6PmW9V29qEVUh7fwfBfF9GsTBZWc6HnziAu7ZEO8cFi7j3oMmK/iKqivLasPlOz3dNQ0h/TvD1JqJeKxrgNfx1734PwEVKqRZG4zxgUYtn4QTeCb5COLfLup3YM3vh7d7PZx61zOio6aBhCdoAWI9jm4N39QM7sx6O3xzGCxidWFRqBodZc/BarmtfA864v+3D9qJ12/CVXzycOIYjYXBotk3X+zvYTJ60x3ffyjiz1stZWLtoZszBSxxT6RJNL5CLPwc1R4KmOR3zucjZb2K8Q5jBy3ovF5WVFe6GfovaSoN7Md4s+pCJw4zJHLzXfuk6vOmrN0au6wTPd2yh864yeHpfvX+e9EsyTP9/qDr67yVfbSLqtaJdNM8BsBHAOgDXicgzAOzPvAcNXDxwc/zATWfCdEDiOBI565GUksD4fifrNTz9iEOwcfeMn8FLCfAsmULbHLzZjAzGRM3xs1AKNcsC7WbAkjUH773nr8GBuWRWzbzOdm71N/9xp/fzt58LIFqiqbM7ZlZRz72bsWTwEnPwMjKXrj/nsPg6ePb9dJTyshXidf/bPxd9rmsOUKulz8Ezh3jGZQ/gqOUTeO+rnpU4TnsIA7xeyCsfLktnjmzlglX30xuHZRLKZt6iJZrJEuQyqiyr08H9CLwE1AP67+UofKlCRKOlaJOVryiljlFKnao8mwD8VsVjowWKf0PvCMKFzjs5JZoZp5lmcLX6qGUAvOyVbV4dAOM42XPwstRrEizQ7lhKNM0MXFbA9Oiug9g1PZ+4ftoIxPI+ag/Ot/F5f+6GIxKUi5lZRf3BPV0gwGsFJZrJI2fNlWx3VKKsLy2w6rgAMks0ncwumuYJyNnXPoLPXHJ/ynGqm5+0UAsZUlo5nusqrNm4p/xYLHPwgu6WFUd46SWaoxOcF3l/2ZqpuIkumt2UaBZ7nlxX4ef3PVHq5D1sCDT8rwEtHOfgEVFVCp1li8ihIvJFEVnj//sXAMsqHhsVVHMEh/sNT0xppZc6UMoq0czO4IWXj/DXdKvXHOu8OiDsomkGKZONcgFeo+ago8IumvGu0mbA0s3cGHOuXN7J1bnXbQgCSkckzOYZj0/fPmMp0YwPL2sehg5obU778V34X/94eeS6tLHruUe6RNPWZKUnc/DG5Bvpe7ZO4f3fvT34siBtXuf5N2/EW8++GVc/sKPU/sOujslgOuvLlV5I7aI5Qhm8Iu/H/UZWXj+mdic2By9lNzPNNt5+7s342bptidv0a5b3d+aHt2/Be7+zBj/45ebcsWphiWbhuywqF9+1DXc+tnfQw+iZ4G8/c7ZE1GNFSzS/CeAeAG/zf38ngG8BeHMVg6Jy7jv9f0dOCq//h99Cx1WROWGA9+2w10QznMsGJBfr9ubqhY1R4idBZqB2+CF+gOc3WbEJumiaJZopwWCahl+i6Qbr98UyeEaAl9VkJU2kRDNjux0H5vDg9gPB75ESTePx6xLNg81kBi9+gFZGkxWlvHUJbUGePjnQnUWBjBJNN8wO1RxJvKY1R1DXc/DMoMP/WfYpHfYT1LOvfQRf/vnDuP/Tr7fe/jf/cSc27DqITXtm8OxVy1NP5tfvmAYAbNk7U+r4tiYr+nLVGby04Gjc1sHbe7CJQ5d6X3zp/6MdNx7g2Xe0fWoOt2zYg1s27MHrX/iUWIfcYlnq/XMtAMBDT0znD9an/5+3ezznc1z80+UP4MRnHFFpl9l+Cubg8eUmoh4rGuA9Wyn1FuP3T4nI2ioGROVN1qPz2Z52xCHBZTNA03O5lPIXDHfCbUyOSPBVsiOS6CBoy+C5SqXOwTP3FY65bIBnZB4tyyREArzYydG//uGLsW1qFmde/mDq/iMlmhkftr955jWRuYKOhB396pYSTVsXzbQSTZuO3wwn66S/5bqYdGrWfQf7UXoun33tQ3PR+7Ylq1S2bG/Yy/zOuCynPXps7by0Lw2CkrqSxw+Xn0gGG9WXaGYb9tcOKBaE7plpYrVfaKIfU8uNLZOQcl/z5W53FMwpw+FJefYYVvrB5b7ZZu5YNf3aV9Ghcxy47mi8P4sKKh4GPA6ixeQ93/LWvvz2n5w04JFUq2iANysiv66UugEAROSVALgS6wioOxJkery4zW+yYnbRtGT6stbBM7fXa+EdmGunB3g+M7CYqCWbrGQ+jpqg1fbmz+hAxRSZgxdriLFqxSSOPXxp5v4P+N+2e9I/bm2NYPShzTHpoO3gfPai6t626SeMejmLrHP+Vkdh0v+fnFV+50JBgERmF/CarIgI6o4Y80KMUs2SJ5zDeBJWZkjhM6QzP/YgvNtyyqwF5AdVohmU447AEmxZL2Wj5v3N23swDKzCklg38pynva+jjVii27QLZvBWLvH+Nk7NtLI3NOjXvtdNfcaFl4Ed9Ch6Z1yXlSEaZtc8mN2xfVwUDfDeD+A7InKo//teAO+uZkjUSzUjwAsyeIiW9dmCOR0D2E41zfhAl2gemGulNk7RJy3mYbqZgzfXavtdYZnpigAAIABJREFUJZGZwYt/+11zBPWc4HNqNjwJK/NhqxCeAJpj0uOZsZRoxk9Q2sE8jCS9nEVWwN1qu8CkHntKBs8vbYUkA3pv7E7wGGwnsGXPP4bphCUrI2b+P8iSV/Zb9vHaOlYGJZKVf5+fVqIZHccwyxrjoUsb2DXdxB4jwNOBnDcHL9y2SAYvr6Q1jf4ixfzbkke/V7spM18MOkqNxPuzqHCZhOoe0+NTs1g+WceKJcl5+kQ0vop20VynlDoBwK8A+BWl1EsAvLrSkVFP1I1uJAIABZus6KtsJ8e2OXjT821rk5WTVh8R7MM8Tl62L67hOFD+8g5mOSHgBSVZc/BqGfMDtUiAV2JcrlLBt+1mgKczeNOWDF5cK6OTWthFM+P+RvYy7bxQnxQ5fpYuTr8X6o4Ea74tpJ38qJyE5S1gHpRopsyJ6r7rYTKDF3YsLbmrkoK5fvERBQHm8MtKcOnSyEiAF2TwVKwsNmX/Bd77eVltvY99pQI8zsHLomLLXIy6sClVdcc4+fNX4dSvXF/dAYhG3N/8x534g7NvGvQweq7UWbZSar9SSq9/96EKxkM9ZgYd5vp2bZ3RgX2ZBB2M2T54zEBNNzHwSjSjOzr/T0/C+X96UnAiGQnKSk40atS9zJJS4Xp+2kTNwXwnfQ5ezZHIvEQbs4xq30wL59+0sdCJhFLht7At47j6si2DF9eyLLyuuf7cuazFr5uRAC892+AqlVp+q98LaRm88nPwSm3elfjJeh7bt+T7Z1u4b1tySc/48502J6rbYkrXEsyp4LZqn7y8LpqjcAKdlfHQlQTmlz76OW3HSvyKZOfSno+897i+vVQGz//JOXh2XqOtQY+id4rO51yozXs4o4YozUXrtuGXG8enO69WLo0SVXErAIp724nH4mOnvqDUfcygyxEJyiU7GSWajhOe4NriCjNQW7HEq/K1zcFbfeQhWDpRs2bwpOQ7r1FzgsAtXp7ZqEUzeHdvnYrcXnckmA+TxvyW/aM/uRufuOherN28L3dcrhHgmYuaZ83Bi2tlfFsfZPAK3j+ri6ZS3n/auiWbqZ/Tut+tFIidRA9ZBm+22cGzP3opvvyLh3O3DZ49y5D+9Nu/xKlfuT59Lpb/M7dEM3cUse2NjJIWLklRcmelj50d1IxCbFHk7WUL5Ip+KaAi903bf17217u9zBw8XXDBOXh2rhqvJQWyOijT4jQ120qsb0vUjYUEePyb1GdnvvUE/PlvPKvUfdIyeB1/Lhtgy+CFAYUtc2QGak9a6U3++uCrn4NGrERTl2yGc/AErzvuyd64ymbwHCf4MIyPd6JeQ7Od/gcxa/6atm+mmZhDWORbdKVU0GRlumkL8PIzeFkncx3lzZkrWqKZlW1QUP4ah/YmK95PsZYNlQ3Yqv5GWnc9/e7Nmxa0n3VbvC8D4qONP0OpGbyMTHcWa5OVPs2BS9t7vzKIvZA1xCB4tjRKabtudA5eoRLN/CyfjT5+fM3JLOYXcJTkjlmTFS6TQHEnfOoK/M6/3dDz/Y5CZQb1VmaTFRE5APv5gADIbktIQ8Gcg+fE2u1320XTNFmvYeMZbwQA3Lh+V+Q2HTCJEUh+7Y9/Fa2OW/oDrVEXzPtZuvhcv8m6k3kSpTNWn/7d4/GP/32vdZup2RZWLKljtzFvx1svTuGMy8OW+pN1JxjHUcsngyUngGgwV6pEMyOD5zVZyX49zOxl6gmrqzuQpnXR9J7TaBfN8PbSTUQqTkDozHSZk+csrlKoWfKk4TIJ2ccp+9GZVaJZ9Qexfr/Gv7wZpQxeVnClAyvbHNL4QufpwVv+sfJepm6eR/1fM+tvwmLWGbM5eFkdlGnxenhH8bUzi1Kq+iV4aLhkZvCUUiuUUist/1YopYp24KQBimbwonO50ko0vTl44eW4osFfkMEzSjRrjmBJo1b6D03NCFSfvHJJ4jjNjHlsOpB958mr8RvPW2XdZt9MC8uXRN/SNUcw13JxzrUbguuWGAtivWz14X6TFR3ghVnEIIPX9K7LavKSuw5eqSYr9hMF78QIACTyXGr6OYrOwcs/EU7Tr3lkRZpRFHmvxccbv0+vA1ZlabLStwArZf/DOgdvz8FmIhOe9f5yLe9f/Zja8YXO0/Zhi7xj8rJsXT2PwgxeFm8u8aBH0TvBl2kDHgfgvV+/8D8PYus+ztcbR6NQmUG9tZASTRoBZrZGED1x1YGa3kQHIWYQ+MyjliX2aYkPACQ/pOKZtnjnyzLM+Oiph0WTxxO1nADPOFZanLXPz+CZHJFEg4QljWhGVCE8oTxomYOn5+WlLSEBhGsh2egFyhc6B8+73muyYhtKrWZ00bQsvttthqoqZsldL+SV6qUdp9sumnpzWxfNsmsOlpVaotmnEtGyXvrpK/FbX7gmcl3WCDuWQFm/p/Vc1GA/qcFukf9T2c9TN8+j/n/OOXh2Y7fQeZDBG/BAANz/+AF89er1+OD37xj0UKgCQ/AWoz5jgDfm4oGUGSrUYhk8Xc5pBoGrj1yGuz75usg+UjN4sb8gYVCTXHOhaBbQtn0iwKs7mSVN8TJVm46rsGIy2oilXksGeJN1Y4F28U42dDA3bwSZOuAMMniWJSSCbTPGrgp00SySwXODDqTR50PLy+AVCWDMwKT6MkPvZ5lStqwtExk8/z1rNuew6X6hc33c5Biqb7Jiv36YFzrfcWA+8nvW+6tjzUB7P+MZvIWUaOZ20ezieTTnSFPSuGXwwrUvBy/sBj2EfwBowcbpixEqhgHemDMDPIVoFi0onQw6KPoZvNg+4h0oiybf4p04zbvF93HWO16K//WUFan7CstJgSevmIzclleiacYzWYta2zJ4++eiAZ6ZlXREABXOAzM/GHWQNOt3w7IFVcG2CyzRbBZqsuIvkwCxPgd6eHXHMdbBM++ffnyt5ZqBZv72C1EmgCzydk1f68z7mdZFs9s5DbYPW33VfLuTmM/aS8EcvJTjj8KJgKuA137xWvzxN25N3pbRJKgTa7KSun9Ld9O4vPdg3hqLNsE6eOMUxfTQuM3B04+k6GO69qGduPK+JyoZS/h3gRO1xtEY/behghjgjTmzJb5Syt5kJSjR1Bm87D/wabfnta82b43v49jDl+LtL3ta6n31WFetmEQ9VmM4UctpshLJ4KWPb0UskFUq2eLcXArCiWXwzABPX2623Uj3UpvMOXjKC0qzMp6tdn5gFSyTYDRZMct39XMU7aJZLiMXLRXtTwaviCKbps3By8vgBcco+XCzNj/9Z/fhHd+4Ffc/nlyfrxfSjh1kE0bgREAprxHBDZZAuBN7zebbHUzPeaXSiTl4BbJz6RnPvDF2X6LJDF6S8ucRj8IXEEWV/T/37m/ehj//zppKxqKHUHL2BI2IMfpvQwVVFuCJyNNE5GoRuU9E7hWRv7VsIyLyFRFZLyJ3ichLqxrPYhXPHJmBQrJE057Bi0sLNtL+gEjeBvC+NcwqY9QPY1Usewck5/rFRebglcjguUolSjT1SdtznrQcAu8kT2cPm5aFzgF710pTkS6aeg9HLZ/Ay595ROr9s1q6exlcCV/3yNzEMItr66JZ5Hyz09cAr0RpptI/M0phU2JsHSykZvD0/UsWWdnGoh/Thl0HASAzK70Q+tjxEQSdPYeiYCxb1nOjH4d+Pn/n327AFX7WIz4HL+19HZ2DF142A6+8DF13XTT9DB7L5BJsnWdHnRqi/3PB+5ytFsfSOH0xQsVUmcFrA/iwUuo4AK8A8AEROS62zRsAPNf/9z4AX6twPIuSGVzE/3+HJZTeT52dypsflxavpAZ4uhFFxj5FkFgoPXpMbye2Bcuz7gckO4mmWVkgwGt1XBy1fBI/fP/JfpMVFZxstlJOOvMaymSugxcr0Txi2QQueO/LE2PSsrIN+g98LRbQA2GTldQ5eJmPQG9jZkYK3GEBSgV4BUafzOB5z4c+0e/kNFkpyzb8+HXLJmvJjXogI8wFMDzLJHzj+g24+sEd1tu27J1JvV+8RPOhJ8KW4822G8tMp+wjJQgssuZkeL8unkj//cQSzSTb3MpeGVTZZ/jl00AOH6HHwAzeeBqCtxj1WWUBnlLqcaXUHf7lAwDuB3BMbLPfBfAd5bkFwGEicnRVY1qMonPwVDSDJ3ob72dQzinZfwzKNkjRNf1ZH2KOSGanSf044lk2wFubLu6nH3hl4r5A9gLr1hLNRICncPxTV+KwQya8JitumLlLK7XMmn+n95nGjTVZcSwLlReZg9fxVjr3umjqYE6Sz0tqF03rnDEV+z067iqV2X2RbVOXl/Cf2rxmLuVLNC3PZ58+gvXrFv+fEM989dNcq5Moh/7MJffjT771S+v2j+1JD/DCLprJx7F9/xwOGN1u057ztEYszQINjcLbM2+2YpOVdFU2IRpUgKXff1UdvkzgmvZ3gQZr9/Q8Vp92CW5a4LxsZvAWn77MwROR1QBeAiA+I/4YAJuN37cgGQTSAtRiGbzIMgkFSjRt8VDZrEWRVvKOk56Je9Exh+I5q5YDSAZhgL1E80XHHBpcNp+D7Dl40eBRKSSarLQ6bqK0tdn2Gqmkfeuel8HLKjcLMnjGvuLPf7SLpn0/kRJN3THTmJ+pY9Ay6+DFH6+ta2FV8j6s9s+18MT+OQAFO4BmPG9AevAeZPpyjxDbr2V38TFUVaWX9nQEpZsDOA/4w3NuxgmnX5G5jfk66gDvqOUTie2yMj1KAfdsmQp+L7IEgrmbSBlyzuvTTVdZ/T+dGbykcL5aBRm8nu+x4HErzuCV+aJAb1n2C1yq1trN+wAAX79+A3bsn+t6GR3Gd4tP5QGeiCwH8CMAf6eU6qprgIi8T0TWiMianTt39naAY878Y61iv+vL8RLNvL/vqXPwUj4mw3lK2fu0LQZ+/FNX4md//euY9NefWz6ZzODZMn9mUGWWImZ10YwvdG4v0VSRReBdpYJv9dMCtbw5eJklmkrBccIMXs1JLpnQLjgHz1V6HbxkBk9nGeuOE54UGLuy7TZ+8mD+NugmK6/94rV4+ed+AaDYSVT8pFE/M8E6eCkZvOC9XTqDZxtD9PeqnsO8ssRBfNO7zgi60pjvN/3/ctlkHR1X4UMXrsXd/j70CVA8QD5ymRcM6hMmAKlPhoq89+3/v/Ln4JX/wkP/l2QGLykM3Hu/70FlN8JlEryfazbuwUNPHOjh/kts62/M+G44bd03i5M+9wv8+9Xru7r/OHWfpWIqDfBEpAEvuLtAKfVjyyZbAZitE4/1r4tQSp2rlDpRKXXiqlWrqhnsIpDoohmc6Hu/6xJNRyS3nNK+f/v24Xym9H0K7GvF6Q+oGX89ufg8OaBck5Wsbyfj2UFXqcji5YCXyQkXiPeep1Y7O8uTt8xBWvAA6CYrEgSVtvHnZfAaNQm7aMJ43S2ZzWgGzxhHTgbv9k17Meu/RkD1HyZ5J2RP7A/XTSsykrxlElI7nXY9B88a4sXGVFGAl3J9WAI3nCcC5vvNDID2HGzix3dsxVvPvsm7LSXT86SVS7B8so4te2eD6xS8gO+sa6InTWnBWbS7ZvESzaIBm/7vzbXIkqr8AmJwJZrR47/17Jvxui9d17P9l5ur7Mnrok39pV/Cx6e8ipRrH+ouydGrL0Yuu/tx7Jqez9+QBq7KLpoC4DwA9yulvpiy2UUA3uV303wFgCml1ONVjWmx0yf4WjxosGXwbH/r0xJSJ64+PPv4GbeJCE5+1pF4/fFPCa570opJ/MbzjgIAHPDbnMezbECBAE+SgYxNokQTyRMzs0RTxPsA1Rm49Dl4OSWamcskRJus2Mo9mx0Xf/m92/GZi++znnTWHQf+FDw4ItZlEvSHujcHz++imdM0RXf627pvFm/52k34yI/vDm7r9bfsH/nxXfj0xfeF+y9xgCInOXnLJGS9RkD5+XO24ceHmTfs1addgg9+/47U2699aCdu37TXcmx/x/G3kj6BHoLYwhYQmdfZyojn/Qy6Hn/8Na05+guM6Dy63/v3G3Hm5Q9Gtk37ckOlBH42aZm/7Pt4P5nBS3IrzOANqotlWF3Q32y9Tdr6mDRYQeC90P304D02Pd/GX15wB9513m0L3hdVr8oM3isBvBPAq0Vkrf/vVBF5v4i839/mUgAbAKwH8HUAf1XheBaleIAmkWAnOpes4ZfpmQud2v4mpH3D9+eveha+9Z6XpY4hcw6eAEsaNZz9zl8NrrvtY6/BR97wAgDAAX8unG0OXt1S2hnZd8FlEg5bGm+yohJzYcwMnoj43Sn1bfbHV8sZX3YGz3t99GtiG3+rrXDZPdvxjRsetZ5INmoCV6//ZZRo2spVa44E48lbC0w/N7o5xn3Gum3dfsu+afdBrD7tksS3lP9x22acd8Ojxv6L7zM4icrYJrXJip6D17bfHm8g9Oiug4my3pRR5V5T5Dm8+K7078Pe/c3b8Jav3ZQ8Tlq2ssI5TmXZyp3NkkizU2b8eQqbrETv7/iZcPP/dN5zEb/cKRG0pc3jy6IDO87BS3LDaKjn+x70W76yOXhlduxvyjl4w6nb+d5aL/+kbNg1nb8RDVwyHdIjSqkbkPOlg/LOJD5Q1RgoSiHWZCUIVLzfdaCUPwfPfr2I4LlPXp68vsB3T3kfKtN+qaSti2ZehiwylozjLIvN73vL125ObNPqKKM5DQCo4KQsLctTdxw0kZ4WyZqD53XRDF8TWxfQvGUSJuqO90GvoiWatuetXjO6aMZOZvfNNCONcOLNLMy97dg/j3bHTSxKn2fNRi/j9NM7t+I3n5dejt1N6VH2HLzo7/EvJdKbrER//60vXINjDluKG097deaYbB+2iUClspN8+37DErhie3l8ahbrNu/D61/Y+8bH1gDP+CKknRLg7TnYTC011e998wsVcwuvhN17Qd2UIDCtXNMmUqLJDN6CpQXuvTDwAK+i/Zb5O6mfV8Z3w0V/Bjmxz6TS++nhu2y+ojVaqbf60kWThoNuua/F52LVgxJNs3QvuZ+sPxO2TphFPjDyArygRNPSZKWWswxBZNuM4xT9XAtWkxBE1pfLmoOXpZmSHQLMdfB01i25Td4cvLBEU0UXOrc8FzWjyUqk0QSAF59+JV7zxWsTx9Un2+b75su/eBj/dfuW1Me1EBt3HSy1CLhKOeE35S+TkDxe2gft1n2z1uuj982/rqpz/LSnIWzZXuzA/3nbZnzg+3dWkvGbb3cS15lZLTPbaAZDB+fbweOLl/F6X3AlSzTt+4d1G3OfeWXCHct48+ggJiurv1gFr2sVGbwBlGgWWY9xwccocR6unwNm8IbTQudG9uI9NshOy2mKfN4uVgzwFpHEHLzYiX7DskyCTdaJTVY2LbPJSs5Bf+/F3uoZxx29MnFbo0QGL3PTgrsxnzelVGJh5bi6k920Jquhgqu8jGFWk5W8tbnqukTT9dfB0/PtLKWjdaPJirKc5OqJ3kB4Aqt/xp9bvUxBL+3YP4dTvnANPnvp/YXvU+TDKP6W1lnnrDl4rvH/qWyQY2/hrzJ/75Xgm/qU613lrUv3jes3BPMsbZodFx1XRZ67qdkWVp92CS7JKB0twvYNsRkwtYwyYjMB3sz4ssNVXsOiSHbMzLKlBGSpwV7O66NSAsMsYYkmvyGPq7aLZu/3WeaYVQWYZUo0mcEbTvoVlNjvpffTg7fYsBUWXHHvdrzyjKtw9QM7Bj2UocQAb8yZ3/rEFzoP2v37P80mK1kfOFkfGraSvPAPU/r98j5U3vKrx2LjGW/Ek1YuSdyWNsftog++Ep960/GR6+LzznRAOll3CpWSAkZpK7w/eHkfonkZvEhGIfYXtON30cycg5ez0LlevFxBQSDB82XPhkqi9NLbr23c0e3iwedCyjjSnrHtftB426N7jLHlnGij+wyefj1sQbhedsIbQ+YQLGOyXNevDF7a/0Olx6Fw1jWP4DOX3I8fZmRh4wE+4GVXAeCc6x5Z0BitAZ45F87/P9NRKtpopZX+ZYc3nzU2B8+43ZYhBNIbq5Qp0Sz6WgbLcgzbmdQQsJWO98og5p2a/2+qOnypUnZdas8Ibygt9GXpSeZ7yP4s3f6YN6Xjge29W1pknDDAW0SUQuTMWWdy9Im5noNm/h2JBz2vfM6RWLV8MvUYtrXsipwEL6QspJFSovkrxx6Gd//a6szj6JPGVSsmC/8B1c+b6Axezh+9vDmCZnOWeLAYlmim78tsAGIbiyMCV6lgoXu9j/icQ73/dtBFM2Q7AdJlZPpnPPgsU0YZHCfn9oPzXume2Tk172RY35w9By96Y9hF0z+GpWQubW5WEbbnM35N/9fBCzMkuqnRdGyZEJN+Tr5906NYfdol/pcICx2btwdbiaY5By/4EsJVkf8zzdh81HYsu+3E5uCZX6i0U+ayFumoCQDfvXkjfrZuG655cEdi28IlmjpoZolmQlaJpleh0P1zNohnO/Je6sMx8uhNGd4Nl/AllNjv5fTi82TYltDR5xi2805igLeoKBVb6Dw2B2/5ZA1A9jd4F7z3FZmNM+ol5sOZFhLg5WXIsrZ9ip8R/NipLyj8waafNxG/yUPOiUXe+Mzs0E2P7Mbq0y7Bpt1eJsRVuoumf+ycJiu2bKJezsFV3msbBPQT9gxe2EUz3NdsyzYnyo1sFx+a7SR9of7+v9YBAJYYAZ7+I//Y7hn88TdujQQlyg9szXHaJEs0PT9Ysxm7pudTSjTTA/PcrKLl5vh9BrkOXpFsts6ife7SBwBEX+9u/zfrAMeWwWsZmW6z26T5/2/eeJ+6SkVet47rVTCYGfPSGbyMeXX/+N/34q//406851u/9G+H9X5Zgi8U+pTBG4aOqUVlNVk57hOX47eN+cFllZmr1itZXxb0SpndppXa06BFX5cy75Rez/Mctr8W+rN/slEb8EiGEwO8MWf+rVZKxdbB837Tf9h1RseR7v8Y2DN4/jdPsevf8MJwzbuFfKjkLZMQHUv092WTdWw84414w4uOLlyaYs6Hc2MlYv8/e18eb0lR3X+q+977Zt7MwADDIpuACCiiEBBxYVFxT35Ro4kal5hFk2iSXzT6wy1Ro2JwjRo1alxj3OOCrCIgguw7DDuzMjD78t6beffd212/P7pP16lTp6qr73tv5sHc8/m8z7u3b3VVdXV19fnW9yxi/2pALwVov7hlDQAAfOS8u+Cws86D8W4f0sSMHQeLrURZyvDklAuq0kSVSlFx/0MMXjtNKqX4+Z8yCtP4pMvk8JDufPgGYfDqBB2q55EFHa//nIvuhivv3wCXEnt8DC6Dn33i3MPyYq64dz387X/fBL1Mi0DdF9Bju3AfrPPAVe4cn7FZUjp9yiQezfM409PZACFYpTR3Jghwr8yDmQ8gfRayXDsmmzyKpgXSvUFW5M8U80tjmnvqCwmek+0EH7yvXvEgHP6e82G824f71u4aE6f7143BK794VZApRvFFRwUAmOzlsKw0Dx5EJO75vrVjs+JHjGL54M2S5twkGisC6KGJ5tyUQW7LTM+xucrgjTSM1r27yHBUHuPi5sEzn/GZwIfEKPyDL/BKKXj7c4+En73t2eZY+Z+vDV96/YnWeYNKE9aQR9FUns/BOkiahCKKZlx5n1CFc5+FHQAA+NXStdWxhAQ+4T6E++8xz1KOJEUpUUWQFa2LKJzY2oIRd9drpJWIyvWYUC/2qVeWnwkfvEpRrrkZFOBhf6VT+3lOdhYCDF6gq+vHu9Dr5zDSsucZsqL4mSr5myamgv2n7VUmpKx/O9NEkzKdWsf5zXLlcSbwHl6zNHfoJkNONhdoP+jczbVdD5oo+/Lg9TzBimI+S4p0iOH1SRVFcycweBio6P0/vR1e8JkrYPXm7bPeJpePX3AP3LRyC/zu/g21ZXPzoMy4SLfnBZ+5Ap7xsV/PfGOlWD54s8SNNEqTMGTw5qSY91pzh2/bDHj6c2yO4btqM7rTGkIZSWYtD95Q5p5osJVwBFVoWoU55pSankPvP73oaPtAZVrgXx2mxeA1iqLpLxt7zTTReYzi1kpVsG6qWO4h5PlLS4CGn6ksWdixmI2JKReIYT/R9A7ZpQUjLbjkHafBurFuVbbTKhg8zkiMCQxeP9PwvetWes1Hp8Pg1ZkIUuVovNsDgHnV/KLdyHOy8x/ojpQzjUovKwAeZea0NoA00zbQ2LK9B4fs7W+PttbPc0iTVIz6OBsiPYe2z1kcg+cCPO1lB2OlyilZs8mAAMgJsmIBOm2ZjWZaQyewGWQHvagHcroGwA1koonAdQZ98LTW0M+1lcKG9ufaMmDR5okeHLzXjDU742JMNGeDOTZ1bhzvwrt+fNuMt8GFm89Nx4fQ30Z8WTOH5wbC++61K+CUI/aBJ+zr5tZ9rEpo/RxEJ/NZJQwqc82kGy00hgBPluGoPMaFrglUcQMwYAF3QUY7JsjKTD7HKsI5eFoMXgMTTc6A0WYbR9FUcSYw9VE0ZQaCno/t8LpGWqnN4AlAbLSTFsp32WcEhAs6KRy53yJ41hOWVGU7aVIEp4i4rpWbtsN7/vd2OOt/bwcAyQdv9szMKJDdMF6wZc5OJxTgqWKmMM+bMBHr5vtUllusIQAyeAge7WAfGKTEJ3ZuNexEsz4NKiKDRz6jr6bQJUscgDcDGgSOoeS/STcZcPwyzuBltg9et++aaFp9JmuiLw+e9nyuM38axP9lNhi8D527FJ74vgus/qwfN5s6c83syif4nMwGuUmr/OLlD1hm3rMl1vyB2WFtG5lozjEG730/vQNe9rnf7upu7FSpW5unU99MPOdNavjhDavg1lVbpt1mSCoGb2iiKcpwVHYj0doGUqjs4C7IaCeFRDEmZgYW+zc88/Fw5H4L4VUnHuwtM52XSpMgK7yoBepio2gm5tyYF2htFE2ihErBTBLCFHKWbKSdWIovZTkWj7bh8689AQ7bZwHkubn/yEKNCkFW2uVOWCg3HwrPkTY9JDsKAAAgAElEQVSTDF6dUCZtQ6ms4vuLDje10DTmlG59oZefAsPgUdGsTsoQ1jK75GcTtdRlxGZDsM90LeCmhz6zaipcIe1lGrZs72Hlg/UNg6yUaxKtZpyAZow8m/EomsxE0/qeuxtJWpt56/PN8ylJ9NkfxERz08QUHPvPF8JNZahvek7IBy/GX43Kt65eDgD2/epbfW9U3S6TkA/edIVWeftDW2e8fkl4moQmYCxWGploVj54M96NgWWy55+cb/n2DfC13z64E3sz+yLdLX4Lm8ySzLOODSpN6nj3j2+DP/yPq6bfqEfWjU3CbatnF0A+2mUI8HYj0cB8zsovuMs9v51WIfmfflhhX/anzzh02u0etHg+XPKO02F/IYed6cvgb5V2g90bxwdPyZ9DkhAfvKg2EwXHHrin9/ceebH7AF5loskutZMmXgZvr9EO/MHTDoQ0MWyTAsPgSXnwcCeMKsY+AM1BIC/WBODdt3YMDjvrPLj6gY1R5WndG0oTU6Og2AweVwylJNKOiSa7ln6mHQZP51Qht4FGna5GwZwvgfNsKHwA9QxmwfQig+fvA+/fh869E978zeun1bfKRLOcW/R5pfPcB644Y0eZwILBs9srcoMWn618lD5QZylMYQBHHw9J0V43NgkTUxms2mR837ALPjbnJzeuhqf8y0WNgqLgBpPlY2jVv+sZvJge4BjOFL7Tnnt5hwDw1o1NwkV3PhJV73evXQGfvvieZu2DnpXk9jFLSJZr+Mgvl8LqTUUAq+lEtJ4piTEFvHjpWvjIeXfthN7sPJHWCWN5Atb/pvXNhHnlXGL7X/iZK4z1zi7uy1yVIcB7jIuV6FzbizcqT6gEzWunRUh+BXDAnvNg+cdfZpnvzabsNAYvUDb2xWby4MW12UoS+NxrT4Af/fUzxd+p4rVjSjLRNAqkY6LZTmwfPPIZ+5cowzQqZYLpLFnUcdpCW/YpFo1QkinmJ8RNXJukSbj6wQLYnX/Hw1bfueyzoAOnPtGekxvLgCaGmTK/ZdoED8G3gMy2BDqnfAyenYLB8neo0azoz6jMu2kSglUMLFK1dPdeR/rgcYX0l7c9PO2+VSaa5UYHfV4pU03HmjLgPGom/86fcWqOGuOD5016XsPgSb/3CQvJr8vng4emg/euHRd/lwSv2fdM70qdrQmWMBshM9NhbiKJIkXA/dOvXgtv/c6NUWvaZXevgwsjwCBnV2bS79LUW1/nzSs3w9euXAaf+tW9ADA3GLydEGNoTops6l3+R6DXAM7QdWcmhnRX35azL7gLblhe+AxX1iIwt4DnXJIhwNutxPbBQ+UJX/zzOyWDtwucrKezazidICsUAMfWgnVEA8JEwcKRVsWKAgA850gDUqiCul0IkpIkikQ4K9p8wr4L4LUnHwKdNLF2+yeIclIlsk9MovNEKXj3i4+Gj73iOHju0fs5bSHA8/nP0bHmJpo+H7xVm7bDnWuKXfGzfnIbXHaP69/Cr88naaIcJm1D5U9UghTym+1kjgyexGBxsGp/nso0jLS4D57N4IVypLntmc+o2PEzZs2hPaBEpKpIqWFMNOMZvEGkn+U2y1ZOqW4VmdWUHfMweDTX3ZTFUvG0Ce78Kp6J4nPPkwDd53dXF0XTBwZRcGPHAnjIUtWoUk2WS3xmpzxRQmMjfO5qmWkTzSYgd0XJssY0zSO7+sT24dSz44MX0WHe17mQJmG2rBfq5GWf+y1855oVu6RtgPDcHmTa+9arQWU2AgE1ka/9dhlcTCKMozxKlrCdLkOAtxsJVWYA3J3dee2kYIh2wfo+nXdKKPE6F9cHr3kfTKJz+QQOOCUA+henHg6XvOM0ALBNNCX/mlS5QVZ+/c4z4OxXPtUBHVY/y/6h0o4mmqOdFrzuGYeK/UcTTZ8PHo1W5Zpoyj54p55zGbzsc1eC1hq+f/0qePM3XDM+1K3rAJ5ShSkxlfVjdpAVWkeWk0AoyOAJO+Wh95ZSqmDw2ozB03bdWY3Cb7dnfq988DQv4z9/OuCvYutofWDmV2wUzZBCGvs4//F/Xg1Hv//C6vsvb1sD7/7xrdXcsUw0J/vV/KPsIQVmPNE5BY9aa8FSwLB6/jx48n2lZeqiaErzocrlJzB9vqEdJNR5yjbyfG3uavmnH90Kp55zqff3kA/tIOJLeRF7jk/6mY7qI99EmBUfvAGsPn3P7s4EXbuKkblzzTb4wM/u2CVt+0Tz/w2GZjaZ+l0RUbOwynBZ9LkW3XOuyBDgPcbFVuLsI1WQFeaDtyts8Adp86d/+yz4yhtOdADUGUfv6z0nrcAZWP8B4qNoViaant+5T6BkQpooVYEzyoSJAC9RFQCSTDR9Qn0Fs1xXecBCUploegAePZ2baPII9JwFlPwLUbQAOiRJlIJ55Jr3mNeCjRPcB8+Ux+sGMC9IWRkP72IXJpoSg1d+dhi88HXQn32K00zv5kptm7aK/2mi4iM+zoDCd9NK20n+I+fdBT+8YXU1d2gLY5M92GNeGwDs8aXApUvmrdaMldNCFE3C6tk+ePVgT4yECvVlUQyDZ45VDF7NTWiyWraqTRsKVM3vc0U3+vGNq2FV6QcmCc63meqvnSPMiLhGNgCX/TyPZPDs9rnJ87UPbpx2oKoYoMRLSO/in9/yEDzhvefDio2DJ5NvInNlTu5sEX3wmO9pk7HxmZoPKr6UNDtDMFcrBt6ha/kc2aOaczIEeI9xoWs138E2wQWKp2N+O4VU7QoDzcEYvBMO3QteeOwBFsBb+uEXwdfeeFKgHdu8MibIyp896zD4yMufUn3H5nygtM3SNkhpHBJlABhdnKR8c1aQFdZmKDxw1c/SxFODrjW/kYKsnPmk/avPPrYEwH3x8BdAKPofB2fWxgSpWIGd5Hy00zKJzrEMB3hgMyV1EQ+5KCjMaOcJDJ6VB6/By5ReU8/je8PreMq/XAT/duHdxfnB2sMidQ3b4gAo1M5s+Ayh4AYDbWO82xfzRFImmfrcZdoOXJHl9vxPFJTPBDht2WaZnuM1Jrm+82h/iv9u8JOZVHDrfPBmiy351u+Ww09vXj1j9ZkgKzPTX+5z2cty+OLl9zfL/Zhr2EhSTgAU8ygG4IWiaN79yDb4k69cAx89b2ltPSEZ5N5Kr4jzSv/aux6OD+4zHdldfarCl918TGYzimY3EOF0NgTbniytMmiQuJlI4u6Tv/zW9fDNq5bNWv2zKUOAtxuJBmAKjr2Sz+ukkCThpNyzJdPywSMAarTTCppsphU4K/7HwNlOK4FTjtin+p4kLjjk5a02PQye5Do4Ptl3olsWDJ6sgIcYPCxbmGhqy7fKJzzIyt8//4mWUj0VYBu5UjPFAhJghE8OlIpz7T5TodUqZfvgFSaFxWd8AdB7Wpholl/K/7IPnv2d90LywdNgmBsnyEoDH7xYBm+824cvXf5Aef7gL7RQonPcLMHvoeuYTZMtBFu9nM63DBYJAM/rg6e1A9roBkkrSazAU777Z0XL9JloSpsGNWkzpCArec24D3LbxSiaAkM50+v+j25cBT+7eU1U2Zjros/aTIjtAwfw3WtWwDkXhqNfcnPWz15yL5z4kUtg3dhkdaxPzMKj2wfbB29TGTjq7kemB6hiHlHeVeldjMd2lincbgvwAr8NstzyOT5dofdlMhBwaDbmCbY9WW4s03fBbDJ4l9y1Dj547vQ2WnaVDAHebiR5ri3FlSvTu5LBmw7AS7ltYKidymzRbc/XhUQpa6xSYvooCZpoIpPXEvqnwGXjAAolfs/5bafPVRJa1mgn9fvgVWwlAkRdr8Rh32mQCx/rN8EAHvfJ8zF4CNB+98CGCrD4GCQAWwFWCmAeAdCFSaHN0PmiaFYMnuiDZx/jl9zLcpjfse8jT3RuKfRlnz/9q3vhF7e6Sq7N4OXiCzHP5fJF207xaJHevZqNvw6URYkJ637jik1wzYNxqS+ooDkvbb+f5c7mCYA973gePCvfGzPRTJKiDLdkwHNRfGxenQlmtImmABS9PngDgDFuiu/r20zrZHlen0+zyaqf6fDYNBXuA7c9YEJencMG6aI7i4APCMgABjTR1LOzYTJTDB6+wnYW7NoVJndzwY8rbJZfPqdN6pthpp7WEGLwZmcul+2W65jF4DW4tkHv8w4huu5clyHAe4zLk1n+tZSwXRzktNOkzIO3K3zwBj93kCiaoommR91oJcoCYxX7Z7EB5HM5xsj4SKBFKXucscx4twgkcek7Tze/KQWLRwvQt2ThiFVPkMEjpqS5Ls3RalQqVKBxlyxVfkZ3vGsveA6Dx4BLxeCV4/K6r15rTA6rcu59oS+mRCkYIQxei/iM4X86rwuWxAaAch48+zsdJ6UKcDGPM3iaBG7RsjnM5359H/z9924Otpd5ou6980e3whcvv7/ss727Px2TlFAePCfKZKCe0Escq/mjL10Nr/nKNU27WM0/KpIPHQBLk0A+a60t/9bCRNOclypV5sErffBqGK6iTvpZw4bxLqzZsqM29YaYJgHBHAOhWHdY4te8VGDwpOALM82a5KXZ40zWR//PVH0A8SZenInFOviaE8fg0Xtgs80ztc06iDIrbn4CuhPsHBC0awJ47PQmHZE338r/1ff4jvr8TAcVa8MtsME3G5F5Qwxek+YGDV629OFt8Y3MERkCvMe4/MPznwiffPXTAKB4wKnPlmTJ+JenHg4vP+GgndQ7I9MBlZKPm08MwCvbtfrgOSdRVgARCRwiMEqUYebwmARAE2WD2lECWtopZwwBXvP0Q+ETr3oq/NmzDrPqCfvgmWvNy2AjdWTnSAXwSgYvkU1JASQGz14ctbaPjXX9JpohMzGHwSNjhSkgAGQlLdeGWcO1WzanC78hepmGdstl8Cr2MG9ookk+9/Pc+0JEkzGq/K3YOCG+0P7rymVw6d1uCGl+bVJL2F9uookflq7ZBoeddR5cX+YgKvod/1Z93VevEUGbT6SyWstsuOQXitFAe9xEkzPx2qw9fQv0yPeSpxc46SOXwLM+fqlzv6l/ZnGee42GwSPl0AfPLV4eb644VWkSahi82QB4PBDTtOqbYf/EjN2fEKjiyaZ5HXSN5GkS7ls7Br+9b73bPp1vICvM073UQfC1NAoxUXVnUnYF2NpVqRmohADGIGM/0762sUFbBoneWicG4LkMXpNrCyV/D1XT5P01V2QI8B7jkiYKTjh0MQAUk5dGeETF5pJ3nAZfeN0JAADw5mcfDqcf5Y9CORdFUvp8gpdf7VJG5MFrJT4TTZfBSxNVATsES6IPHotWOjpCAV5i/YYmoq8+6ZCBomiiD1+RJiGOwUMTuUT58yLW+eABMJ+9ksGTUjuEFmiqeBfRR801txLlmBPaKQhokBW3Pm/7jNmdynInOmrB4BnluC4vmn0u6WOma1+IPZYWQBquf/3lUvjzb97gHOfgUToXu8tNgLHoVfdvAACAC+8wCZybKES/e2Aj3LRic3R5KeIqN7FEkXzwWomCPHcjY9LnCkEgVjmtNAns/mV5fdAd2QfPBVv9LIenfehi+MmNJmBJk/0wvKe2D575XQvHZkJyDd4okMs2TMDf/PeN3mi9vvqK/9Pr6Cu+eBX8/JaH4BtXLa+Oaa2jxpTPeWMya/v90nIv+MwV8Ib/us6py2aDm22Y+GS827f8AeOiaNplZPeFncvg7QofvLng9ycyeOCuCbEy0z54tIrQdI0x3/fJVD+HZ539a7jozkes49geAi36Pm7G4Pk33kLVdAM+h3NVhgBvNxBklLTWVoRHPH7kfovg95964C7p2zmveiocse8C61g7VXDQ4vnRdTQx0VSMfVPCb1xSZqJZpVogZRAYKZL+YCTA4CmwX6QY/h3AmMqihPwTKYPHg7PQaJ9aY5Adb1VWfZMVwPOzfpzBk1goGmlz644eABTBfLhwlmliKqv6QH9TYI8nKukAxJSFKuc5SZOAJpoxefDId3xZdRhTrLXxn+JKndYAW7ZPWeXvXzcO3yijcdmmLjr4QuxludXnXpZbShk3ReTiKqXu9WN9FYPHlApkyWk7TRXSJsq85O+Qay0+S1aahB7eq6QIssLAE9840WCeL18QEms+WQqTH9DnOsZE0050XuRVxPKm3MRUBlt39OCDv7hzWkFWfFE0jd/fzCq4IRPN9/30drjgjkfg+mWbxN8lGTRNwr1rx6q1RGsNN6/cAnc/Mlb5/wKUa2OgDh/7L5loFj619f2yGby4yJt18uLPXgEnf/TXTv+CElEEH7udx+DNHYD31SsehNM/cdku60O1FgwwJCG2ahCJ3cScDoO3o5fBmq2TTkoObBvXMd8mXJ3YGyvy8yzJdFOW7AoZArzdQGjghLZlornzfe24/PFJh8Cl7zzDOnbXh18Mv3nXGWJ5SZqYaCJQ40wFgP8FnybKKs9TLQAYYJQqVYHoTsXgCUFWlAJFDh9IAG2HAbzQfaL+aDzCIGcab165pdYUts198AI+mZzBk0AGVeo3l4BnnhAog4PD8257GJ7/qd+Iv/Gx4cwdZ/DwRYBHNzPgxc+R6gAAJzqrBm21SV941y7bBMd/+FemrNZw5qd/Ax86dyn0GUDr52EGb9Wm7dbYfvjcpXD5PcbkK8s1bJxwr4n+bl9b8d9OoVL8x7H9z988aB2vcqkJwCBWmqRVmBRepnkuPwsUHOMuaystgH/fyoNnm9Lh3GnC4FkmSrlcBr/7/fiKLz3G4EmsGoDpX67NrGmycleJzimAHcBMuankuR/gVVEZqyP1bQ9iSjrR7cMLP3MF/N/v31KeWxzna1Usg8ebllI3cPbWJ3wTwfLBq+mLT1lfvdnOIziIoi2ZASc7mcHbFWSabz376Pl3wYqN23dKH6QeGBeEskyDsQmxVYMIbTu0/k/HB69KPcTmri6/082avcrYBE2a823YSfXQ52xn5/2bCRkCvN1AqP08jUK3K9IhxEgrTYKpDpzyjaJosu/UFM8zHqliDF6NDx6CaGTyfHnwKPu2/x4jVV/aLdt8UwKjKNRckbKAtH06lHW3vMOiaKpAVFXO4EmMDo20hcBKYiSld8VDWwplxWLwlAvwQi9AzP9XtFH838DyVhXn+AEe7tyNOD545ryCsTHn3LTSNkdcuckoCBR0AhR50EKK07bJvjW296wdg7/97k1Wfeu2uddEf6cSUiJ8mwntRGLw/C88qZYbV26G5/zbpRWTG5JJD4MnmmiSF29lopkmkGtXkZfmDj5ffh88cz5VXEImmNQ/k5c96v0XwD/+4Jaqb1V0V0/7Wqijic+y7IPnlpsNE82eRykaxKdrkGifuI5ds6yI5Ipj7DwTkXU6QVaE1A29yDx4HNCL53iqib1XgwAy6ZSd74O38xHeHHDBCwdZqd5z8R31rSmDis+yIdRuU+Ebtrztyb5ZN3HzvMmYhKIb83rozzs7799MyBDg7QZiGDw95xi8mZAm12Hy+RTfqaIUMtFMLMXQLY9gKlGRPnhlPjcaHRPr4CaaUjoFFArYOYM3v2T3bPbRW5VVH5rIpcpvIjoxFY6iCcAZvEKx7+e5wxaEXuhcsaVjY0fRdHf4+7mulCQ8LAE83nX6HRXjTiuxxltrw7xxE01uzkF9yrgJVy8Lv57GJ/tB9qufa1g/Pun9PcpEszzE2WYOLmg/pHQTIfnS5Q/A6s074MYV9WZ5kg+eN4qmFWSF+OBpbTGOACCaaOIhfxRNGeyFzJUyxsrSCHC9TMNPb36oAhmSiaSlSCGIGFC/wOe/289h+1SxKSPtsA+iAN6yaov3PAyy8u4f3wr//PM7rN8G8emaCVNSCUwXx+uCrNjn8/qabNjwcwHQB08KsuIf2xiZrt8Wys6OolmHD2Ylz1pNozsjCIuYLodtYDYRew0asFNELAYv9M6eRmP+Z7T4jm4fuTa6WJPmQkV9DD0AQHcGIwLvLBkCvN1A0uplCpYPXpPcc2c+aX/4k5MOmfG+zYS0B4iiiQtpzJktFtWy8uMjZTDYiQbDguEx0QevPIR+c0sWjlTRJdtpwnz+/H2jrNJCBvAw2iRXakMywkw0myS+l0AI3fXaVjI3vUzD+JRh/97zv7fD1hL8SS84utAnIQav2vkj52rbjBIAYMP4lMDG+UFQt2LwUuu8L17+AFxYOoJzkzyurNGvvcwNyBJSnK5dthE2CWal1flZHYNX/2IyDJ59HLuFkcsoYBo0KEQM+yQDPHAC3QB4gqykBfDnqRHoZkcrVQDaPBO+PHhUI9BMKTflXcAgRce8d22RuDpREAR4WgP87v4N8Jt715P8b3JwnTrB9ef9P7sDnvzPF0EvczdYAJqbVV105yPw8v+4Cn50w2rxd62LzYwf3rAavn31Cus3RcrQ/8VnuR++nf2Q5Gydx7F2Iv5CnImmb2OKrlH9WAaPTbImEQ9nEuC5gSbccyofvKhWpy91YGsm8d1Etw+bJ6Zqx2omU374ROpBzuZ9k2v/gy9cSeqe/qBZAG+2TDQ99eNXyuCh9dYglgD8s1QP7UJ3GEVzKHNRqFJDzQKbALyvvekk+LdXPXVG+zVTMhCDV36PGYLEY6JJm0XFP8sNS1r55XkYPHrePgs7FYPXSRMxLYMkFHAs6DAGr0zMTc+uu1zsO42iKbW/gARKee3JBfCXgMRUZhZFZAWzXFdgDwDge9ethO9fvwoAXMULy5v+KyfISqUklscs0zjCjlEGb8nCEev+xTB4I63EStFw++qtpLy2fAY4g8ejIlLpZeEd/89fej+84b+u9f7ey3NYP2YAnvNiZLfFKL1mALwMHmGeeN/DefD8My3maRVNNHMtzkVqBojj3k4TyEofMJq/0PLBU8XmAD7PPgVbYuqU4sftPvnSJNz9cAHwjtp/UTWWGQN62ObrvnYtvOnr1xkQQZlsZxT8wsesl8mJuJvi9QfWjxf/N4yLv+P4y30q/lfpByL6UZcEXhIOFBBU837FKohuHjz8b298UPPtmLqKiK/295DE9jeG9a1TcgGoiebOgXh1zcwkk3jGJy+HE/71V7WgpEmQqEFFDLIyALCTZCaGjAf38klTf95bVm2p/Kd9vrbGN09XVjCoW/Gy67ZNwmFnnQc3CpGbdYDVDPniD33whjInpXrB67kXZGUmRNrV90mDopW0WB68KngJGT8EZ7nWlRlfKA8e3hIEDPssGKkYPzcPnv8+4Q7WfotGnJQJaKJJTSnrAG1apoRAxib1MHgLSMTOlzzlcXDy4XuLildXSELdy3KvH5YUqIXnwaMgJE0S2DbZg29etUw0ZckI64Evpw3jU7BkYcca16APXmZMNCmgpmGTp/o5/M+1hqWQcgJWv+XaTm6d1zMzY5N9729ZrqscgwBC8Ju8XpnFY3yuLt84Acf9y0VwX8k80fs5mwzepBCSupflchTN8v4kyvSvnZRRNDNt5V2kGzUJbg6Uhya6fVhT+n3a7Jz7uajfHOdjnJVKPldAlpWR4Q5aPN8EWUGFhlQhsYPFv+Zjzn2AtQ4rkrGCjH3b4wOda+2NPGdMNIvvUlRPLlQBjwUaOEer9srvoQ0YSbRwj3zHcS7UmxmSz2AHBKpj3WeSwXPTqLjncNeG2ZZ6BnPm2sLNsTowvDOiKMprs/1eG/TSZwIURzN4DW7Q8g0T8PL/uAo+fO7SshG5Dvp1spdZJvt8zl5TRuf9+pXLnPZCIJX3mv48BHhDmZNiTDTtZM2PEXzXCKgqAnYBwn4XKAlLkyClWEAwl2vtBFmhgMScW3zYc367Oh/Bg5MHL3B9e5Tn/9WpR1jsEoABeJQti1GuO2lSKdiJkseIpmTgDCcVG+CVLFCuvYBFYvDoi6nwwTO/tRIFY5N9+OC5S+HmlVuc8llulvOr7t8IN67YBBvGkMEzfZZM7LiMOADPXNv1yzfDz25ZU33nu72cwaPV9z157WKFp1Goy08oNVWZaLL7eNGda2Gs26+ujdY9qJ8FtvDBX9zpLSPNg36uxYBFyMaMtNLKjAZNNPt5bgVssnxplQINhhX83KX3w7M+fikA1LN5aaIskM5f/nkJoiqAx4CFBgIEkMHzsIYS8GuUB48VpmbLVJqyMwiefOtvrv2bAPyMkNmU1L/Yqef4RmoZ4Gkdtzb61gnH71doO9Q3nKu8Hd8tiWbwIsq5m1tuGRyaneCGBgD1Jn6z4Qs4F0w05Y2X8gNuVA547TMxZNa7NVAhf+5vWL4JzrnwbrEsuh/csWZb2YZcP/3e7ec2wGN1YlAw6Z5JG3YoMcHWHk0yBHi7gSC+0MB88B4jCK9JHjwnPHfEqb5E53aQFQR4AJ2WbX7ZYkolgLkn//ZHT4XXnnwIHH/IYhNkpRUfZOXwJQvgmvc8H/7qtCMsUzQAk2/OAnj1lwudVlKZyBUmmm4Z6u/HI1tSmRIYvH6We3fDesI26sVL11afE8bg1UXk5P5tf/Slq2Hrjh4sHrUZPN/OPJXCB8+McWjB5y8WO8qeHWSlX+ODVyf8GscmbXY0JmIgHqoLSDtOgPl0ktkCAHzzd8sble/nviiaRe87rYRF0dTQyzS0EwVnveQYOPftz6nmslJoZum2w8fTTo1QfMYgLig8wlrhk2kUDc5U9XPD2KDiYudFs+vi0gTg8dPzXDsmpUX78XUCGLbdt/6G5rSqGCEXyHgZPAvoxj0vrrly8V1S/GKG1GfCRe8hFmniR6dJX5Wqvxd1dTcJSOOYFwtldnai87r7OxvdqAPkvf7sX7t0XcYypVn7TXK8Rddp1e8vx9t61Zevhi9e/kDwvlZ+ueCuCQC22WfB4BHygt073GSXNpisvvE2nO8UVA598IYyB4XS2NScJgQcHk3SJGQ4SpMgKzwXXChNAmXwkG2Qkpbj/0P2HoWzX/lUi8HrOEFWwr08YM95AACWKRoAYfAmKYMXrKqqZ6IMgpIo2USTgkml/JsFFoOHgToy7Q2fzhf1W1dtgXMuvMdqi46NpFxy817BMmEAACAASURBVEe+iHf7Gcxr22P8qYvvgU0kl5z0Huq0EssMNmSy4bycLIDHg6zERd3zCU+UPs7YUV9gCHpf8VhdypFYBi/0Ik+UGoj962e5E0EVwCjrnVZSMX/tEoBluYY0VfDXpz8Bjjt4T7M5A8W6obU7PlN9G4BLO76YZw+Fv/xzXZjhInvIQVyRI44d87BY0lg1mS78fN+GQuwc1FrDio0T3vyQMfVxRiiGGbDHx+6PTyoTTfY9xLCHxAcYsQ/0OeRl3/3jW63UMm4UzeI7fT58vap7fIyZqL/gDcs3wd2PbPNGK6RSKd87CeDVXd9sAM26KneGD54k5hkp/sdeOb+emRgxnu/RJ77fJoVUA+67Uq6DlpvsFYGivAxeqUtJGzn22m77SocA39BEcyhzUihrlQhgY3cUfG5jhoADLBxDOn4IzrQ2u0fz2ikoZcwoaXsSHqI+eIPcJ26iid/pwhRT14KRFmzbUSgiaSIHWbGiioICXyDTKdFEM482d1k3ZkeHXLJwxGY3AyZ7AGiiaa/a3V4OI63Uuvdrtk7Ce//39uq7pECMtBIL2DYx2aAMT5+lRehN00STR+3j5q8xvnL4kqtj9Sk7GKo3HIBlMHOnXNuAHjdQjImmeZ0VQVaK39qWibR5douqXLAzVQa9MSDEBV6YZw9lsscBHm722EwVjlk/zyslXAyy4vHHw49N8LHDOnkAXqzy/qFzl8Lpn7gcbilNouk9WbVpO/FLC4F83ke7f5L4AHCo2xQ8H3bWefAfl90PAD4TTX89Uru0bZzOVhoRVvaHN6yG/75mhamL3mMSRROD/4SkNoBLdd3+Mq/68tXw4s/+1q1LOMdNTD+7MlNRRJtInVnorjLRNCx3M4TnC1IyHbGY9kB9fP3HdXBiyu9Lbtpw10MA+3rQBw/fAXye46MsAzx7HeGAz1d2aKI5lDkpPgfpBvnB57y896XHwM/f9uxZqZsznTie9Cg13UOAt2ThCJz79ufA847ZzzlX4g5pHjy7vbh+PvvIfazvyOh97BXHVcdilJhFIy0Y6/aq8hJDarOSfpaRMhu4e5flOno3lCrOZz5pP/jMHx9fa75KXwzv/NGtsHnCNlnsZjl0mBksAMCWHVOwfMMEfO7X94kvr3ltOw9ekx1dGjSkl+fOTui0AF5eREXE67lv3Zj1uy8PngUcys91cw03C/KaPode/goGV5YoAMXnhDJ4KIUPXmEGSf32UgLw0EST93Wq9O+gCcJXlYnqDdOprHvId6YRRPEgK1nld0d9tYpzEBy44IeCGabsRQhXWjIth/Hnh+54aKtj7gtgTGvRdwavcemabXDqOZdVv4e6yP166xhLfjyklFFBEI1j/d1rVwKA++xqCFtz+IA1j/hHAZ4EVGn6D2veaRoQpn5Tpp7BqwfZYj9AHk/uSzrbUmf9PRvdqDXR3BlpEqw12Z5bTS/Z9S/zl908MQUnf/QSuOOhrf5CrM5gFE32G242b+/WmzniqXURLXNtW6dRwede9uknbbF6eekhgzeUOS8+5fuxxOC95bQnwNMOWRxd/oh9FwAAwCtPOLi2LB8/xF90/Khy2SHmAU85aE/Rf0+6JfvvMQIAAAcunh9s3ycnPn5vuOkDL6iAHgLGQ/YehTc98/EAEBdUhjN40jTheQF9c0li8HqZP7oeFwrwXn/K42GvBR3RTI8KT2599YMbnT6NCACv28/hTd+4Dj79q3th7VY3cXgnTeEfX3BUVL+57JhiDB7pYm+6JpqZhn6u4YA95sHhSxbAdcs2WS+8GB88LBJKqE7rqlNAg/Uo+cUbI9QnFNOQTJV10RQwndIHr5/nltkp9cFLVAHSuDJZMHjm+T77grvh1HMugy3bpyoFN01sM1OXwdMl44jpU8D638/zyocN7/1UxUSmtSaLTRRcKQm7pMzySJa///kr4S++dYO3Xt7/dWPFM3Pp3eu8/Ubhy4VlVu010aSftXicS3VNrExsFE3eT19kv7x6LsJpROhGAGcSssxELq7PAxf3/MWsK02AwE7Cd9NmMAeRujp3NoNngI79PfbKYwKHofzugY2wbqxbMdxcLrj9YfjFrWvYxqS/Pv5+MBG9/QyeScWB9fv73+1loMkmHL91+H6RonJT+5lca+ZXGACVj8I8eK36IkN5tAsqNYcvWWAdf6z44A0iB+wxD37xtudEBZpxTDQFHzxqHtZJzc4/F2Oi6bb7wf9zLLztuUfCQQzgNQmGs/eCTqXoWuHhS0Uz5pYvHGlVfnu+ICsUWCXKv1lAd70MiMijFXyqOON9qEshkUW8iDutxGEmJ3s5TJQ7jFLvRtoJHH/IYvj6n50Ef/5Nv+IrCb0Onvcuy6YXZKWfo7+XgpMP2xt+cMMq+Npvl5n6PS9KyQevjpWUzAklCV1Pnk+DwUNGWRtfV/TnHCEmyvM6KeTajbxJzasVFGPBX+q9fl4pD9RAeLzbr8aunSaWEsMZvIKVpSZEOG4li62NEm6Sb+N1JCJrB2DmZRMFlz9qeR4O5gBgmCYpjxQKPsM4ZzCy7pbtPac+LlJuvqq/PhNNasLqMdfk4gs2IuXBEy0VmPWLxIYWx4vvdE5IQJWuA1bQGHJuSn3wIsCuJLGpGgD8PrpUdraJZt30nhUGr6bRqZ0RZIV8zrWGBOrNdb118ec+YsPFV+RvvnsTAAB8769OqY6F3gF8Ts0vA75tFwGePP9C5pKT/czahONl+9XaVLM5psObRVYUzV3kgzkdGTJ4u4EopeAbb346/OCtp1jHHytRNJsIXSRir58nJTcAT2bwuOmYVJcEiEZaKRy816ijaDQF4giqqNkokhsxAWkWjrSIyZ4SWT/O4HlNNAWn6l7uT4DMZRvxJ8NxkPywqMT4nI20EueqCobRfy7e/0GC+uxgAE+D8Uvo5dqrsLz1tCPgj08Ks8xZXjB4qVLw1tOPAACAG1Zssn6nIkVjw/st7XjytrS2g7r4yvkk1/EMLpeUbDiEfPDmtVKY6ucw0e2zzQjCoCtVRC/kJppZDnnubh5oogy0EmVtXvC8fbkGy2zWMDwG6PWqzzbA6zD/PmkoG/ngcQZPa1GZvXHFZjjtnMtgvNuHHVMmTYpPTFRc+9o2l6abwT6yeqki5gV4HkUsaCqMQIkdd+efFje/+DvCF20Xx7MuWTkFeN4gK4mqBRvxPnj1E8VRpIUyBgDsHIi3K3zwpPulZ0DBv/jOR+Cws86rcmyGREoFMuil4hi94oSDauvBWX77Q1vhgfXj/v5FJjrnz3DF4MWYaHrqoF8ne7nlg8d7Ykw0hTQJuT3GtqU0B5Xms6TLzHUZArzdRJ579H6w36J5u7obj04pVz/coZbMLCUfPImlMikW4ptvmpAelQiq8FYMXsT5NAVCwc6F++SLopkmygI2KAWDFwnwSIoHKbiNBJRjAR6Xbi8PKkYdBvCbCFXs0ERTKQWtRJWsjtznM47eD974zMOCdfczkxPoiH0XwjEHLLLYgZhE57EMXlGfbOJHJRhhTccDfC5KmYi2xkRTAHjtBHb0Mrh++WYrymNK5lCiCiWFd7XywWO+sDT6ZMoAHn/5Y70Y4AXbMAyoAdMSg0fvkRhFswGX4rBOeS7W+e2rV8DKTdvh9tVbq/nq5NAj51VBkzJbmdqKDF5gDvB6KeDy7R1wn7WqfEDR7HueZ74251peG/my5vqr2X2gGyQSSNvhA3hgxpaaaPqurJbBQxNNT8FQNEQ50bnb50Fkx1QGL/zMb4LMcEw7swLwhDrpu8QX+blOvn/9KgAAuLPM8xYSLcxrfg9jQTaeb2Iw+M/Dx/GhLTvg+Z/6Ddy3dgyWb5iAHVMZvP5r15LGzcdgkBX2G1pXyAye3G/+iuBrT+GDl1jnoIRMNHlb1rKi3d9pm482GQK8oQwlUhaUAK/ORBND9ErKsnRunTQFFKh40qiaxm+w/nyaxJyniKDHaf+kKJojrQR2CAt6ruMdljeMm9QFyMRQ1k7ywatb1Iu+pc4xfGkUdQgAL40HyVwsgFcGWVFQjGM/8zN4aaKc6KhcenkRTMTK6xdS4PC/0GZMridkDINlgiaa9ef7BE0rAQiD189BKdsHj46ZlfuTPH8KijHgig934DfHzfxop4kFSjiDl2mfiSb+N2bKJvl28X+klTr+WSjGVNAZGkcmexks3zAh+OCFFWQNGraXDB4fAwrMkeVD9g1/GytTAQRNwtj3Xg0wApiuiaZ9nK8/WoO4KHPrCV/gB87Q8v6iSAwebjTQIDv1QVbCv/c9141im877WQsUkwcv2GytbJzowr1rx+HetWPBcnXtNMV3j2ydhC9cel+jwCAA9jgNvClV/se2v3HVMrj7ERns0R70shw+fsHdsJGk7+FlQoIbJZWfWlQvC3nBZ66AMz55OVx2zzq48v4Npk4LgPprcxm8Ym0OMXhmnLB++VkDKBg8rXWld/BbZzad3E7yjRWbleRlzedHo4nm0AdvKEOpk/IhX1gBPCj/yyaaI8gsCCBGOrdOmjJ4xkRT8sGrr2vBCGXw5CArFFgpkBm8TiuBiSl5Qd/hOc5lw7jxgpIYvFQIBRsDHgofPPvYZC+v7g+vo5WoitEZhMF7YP1E9XmqTJOALFIo0XmaGP8FnyDjRNkp+oLFa/nJjashSUgkRuZsDhCnxMQweCELziyfholmYsa/TYKspMoA4Xaq2CaAG2QlKTcuaHh6lCIRvXY2D3Cci34oK2CAE2QlL5QDrMMFeMbMlSffxgAxKBJxFbOL/5+/eRA+c8m9zvEs17VBPJBp4gCHzo8q4E55jPsohZrgjxCdD34TTfpZi8e50LyDdnv2/cINFy6uiaYMhqSNobogK/iMpAk3FY5IkxD8lURrJfXcv24M9pzfgX0XjYi+0ea7W3udj1asVKbgNfOvbn43ZfAuXvoIfPLie+GPTzoE9tvDWDJ9+mKTY1W6XzSwxnQVfKz9Q+cuBQCA5R9/mVOGXtfFd66FL//mAbeeyEuv1iq2ySSJT8W4eaXNtFrvjJCVBvttXiDICu+WL00CLYdpEvD97/hRszWVirWOMPcIvL7vXrsCfvfARjjrxcdUvz0aTTSHAG8oQ4kUBHjSukbBFAKktkBrDQIOPHmEvYKmBJYPXoSZBsoiaqLpzYNHlWYl+gkWDJ4M5LZHAryNEwbgST540tjURYLEvnWYmWa3n1l57qjQsoPEJvrpzQ+R/hVJWpUq8qkVCrd8Xpoklf+CT3pZXvjgIcBLmAJcTth3/uhWAAB4+3OPBABuDlT8DykxGDkyy+oZuJCPXj5tE83icwXw+hkkiaqewU6aWHPWni/cRNN9no2JpgDwysLtVFkvfP7yL6JoauiU88n1wTOMDSr2NN0D7VLTROf/+sul8F9XLoM3nPJ48fdcayfwil252YDhAEd6tnrMRBMgrPwBCCaa9FzPxVl5AkmRK+/bIJS2z+E1SiaakvBH3Vcuq+5vvIkm/p4mCkDTtdlsOvjuc934SqapZ376Clg82oZb/vmFFnDxMfxUfMEsmorP7NAtV1dPs3ZxA4Hek36Ww+cuNVEj6xm8wa4dpzrfBJKEdqHOz7lOqL9w8d1f1rfxe9tqO20CraPOz5pKyAev2tZggXzqGDx7A82uk5uNW+0F6sV63vfTOwAA4Iyj9q1+G6ZJGMpQHsOyYMS2I7cSnRMl/EXH7g9//7wj4ayXPMmpYxBfhqag8MuvPxFefeLBcNBeJhqnLyGoJAs6xERTyYkVLAZPyX3stBKvzf2OXr0tPgDAI1sJwBOiaErMYSyDN9q297d6gWiWI9MEeFb/SgZPQTGORdAVud1UqSiARwN6JEpZiriTJgFcpRdfegfvZUdwpbLXaKdoL88hq1F2Qvgt19NNk1Bc5wgxhU6UeQZH2qm1I02VOlQiFBQpQxCIUZnqo/KQiMcBit+sICueNAl4T/ByKwZPGwbP8cFrJV5zRJ4bS5L/unIZANhRdKnUMXiv+9q18Itb1wCAWa/ueGgrnHvrGlFhknbLpY0CqljxR4gyeC/8zBWwfMMEcNHCONy8cjO87X9u8l5LvwJK7B6X/Xv1iUUAI+0JssKH2afUVonsa/Lg4TxZs2UHvPvHtwFAMZfoPMy1D9TT6xe7UYnPBw8jnNK56yq8bn0zFY+tYjqFRq55cCMc84ELinQkdQxeQ4RXmUOT83z+lFRmwkQTZbzbjwC20wPQdl3Ff1+uOCq+28vXNZ/pOBc+VLgZJ+kDPr+3EMArzOXdPKMo/cps3O0jtwSwLCTKL88vcxdffs96ACj0haEP3lCG8hiWhfPaAFAs1AB+H7xWmsA7Xng07Dm/7dSRVIttfLtNTTSPPXBP+MSrn+b4yQHEvUCcICtC+wkHeEKZkVbqZepiomkBMBNN5QI8iTmM9cGTTB9971/KhsbkEgwJ5r1LVGFKGEp0nhJmyidT/RLgkeiuVHnxmbpIEdvectoR8Lzy5cZlr9F2VV/d7nKdX9SgylKamOcOAd1kLy9NNGUGj7JrVaLzpHDCu2/tOIxN2koHprHgU7pL0lu0UmWZ+UlRNPNcVyw+NzvKczdNAvqyjbTT2kTeMfqtb4hDJsEo37uuSAiOz9oHfn4H/N33boZflznurPqE3fI60yjOGPDd8asecFm5TJivqzaHIxNmnnQBOOYVS6vtNfllxz2uPM+vZNrHi/91aRKQGb3gjkeqY2miLCZZE7Bnb8LU9wOlzgdvMsTgSSaa5ZoXC6zO+slt8O2rlzvHQwzeFy69HyZ7Ody2emu9iWpDHITzkS5bUn5ILlSpHxzgFWM30e1HREetry02wBLeR57qQ+yh75XGfoidgxkP7FX+l977vvdTXRRNrf36VC8QRZO/96SNk5G2HQn9tCcugVedWJ8zea7JEOANZTeV5ko67upgPkGLwatRwlEGMXWZiYT03BcoJDTICgdyKNQ0UoESTSU7aeIFeLEmmlQkBk8amlgGT2LGfONDTTTr8Dbv09H7L7K+F3nWAEAVu/ch5jBNVG06jyluoqnslyMfj0qRJMfwJddKEnju0fuCJItLgBfjg4dmqJJMx0STBlmhz1xholncT57jkIIvHMp2Upho8gAGAMV4au36n1ITzVaiLP9SKQ9erjVJdG6Dub6QJgGBEvfBk4YxxtTaD0bkNAmS4NqzfxmB+Ze3rXHK/PCG1XDZPeusVAfSzjm95/wZ4fNB2rihRfDaxhk45+NStxaY9dh+Hl5y3AHwltOOcMZp+1QGdz3sBsio0iTU5PObrMzn7ciuGmwGT+o3PVK3jks+eFRCPnjSKXg7YufNlfdvgGsf3OQcl4LRoNAoonXNNGW6cH7RjSlnXRT69JZv31h9HtRvGOfkeDerXTft85od51IxeFUqAf+Jvm7x54nWEVrC+W94vzaMd+FBloYB21CsLK/DyoOHPnjK/Q2AMngCwGN12gxe2Yccr6M48H+OPwje9aJj4NEmQ4A3lKHUCC5qLz/hILjx/WfCUw9eDAC2os/9uXyC5zRZ6GciXyHWEWPWT027qEmclX+O+uAlskI20p6+iSYV0URTTJMQw+AlMCoweL6Xp22iGb4fbWba124xP6by3hsGL/e+YGP8L3sl8EA2ggdZcQJD4CuOvtjK/4nyA8rFpYlmjA9erv3KYMHgNdyCL0WR/lkAjzB4rdT2CaXgC89tpYmXie32DcNK59cUia75+H0WiCkDUJCF4T4ixizTgAEe4Gakbfvg0ftn2Fex604fJIkJsoLCWfOHt06K5d78jeutMPKSYkXHi08xrjxLz7VkHjbe7VllfEydTyrzNdBW/Xjv+en/9KNb4SX//ltnXZMCQ0jL0I4pKQBWyeARc1Lp/sQGlgEgJppsDuB89OXjA6gBApHzRmsQU+Tg6TIrXSr6qh7ANQV4U5kLeLmZeU+4tpWbtjt1NBXczJzo9htd12ArpFsXroV1ga/q+gPQgMHjzHdZ/y9uXQPP+9RvgmW1Z47QLtprtDtWZm2t97mzx1xbfcL/M2WivLNlCPCGslvJwXuNAgDAiY/fa6Dz91k4Qr5RBi/sJ4Xy1TeeBK89+VA4bJ8F0W02TXQuSRMGr23lDTOLGwV1HGSJUTRTf5CV2CiaVLBN2/9P2umvv8ZOKxFNNH0v8ZF2PIPHg+twXy5k7JQqrqmXa68yLkUJ5dIrARdN4k3vs7PzXP60fOME/NuFd8PyDRPVC1gp/3xDE82eJ48alSzA8k3HRDMh/p6tRFVzgaeToPeoazF4CPDk6LAAxkSz8C01xy+84xEYm+xBogCOYqwsZ/C+dPkD8MD6iWCaBK+JZstm8CSgHPMch9INxA4/PnN4vx7eIgM8WgZAZjuofxoH13w+SPcmE4AuZ/D4uEQDPG2Dl0SZ9AWSbNtht1v5UdaYaPayHNaNTcI9j5g0AakqomhSZl26P3Z19c8fgAsE0W2AMngfOe8uq4wEBHBcY9Ob5FqLa3xVTyB8fco2qOT6o7pRCaZ/sfODMoAnzNnFo214/SmHWnU0FdwMmOj2a68rbuMmrl0eZCV0mj+wES9Xfw6AuxHggDW6EcKq8ZloWhtqvQzyvNzwU27U2V6gfnt+a8aMYx/stXkmrKh2hQyjaA5lt5InH7gHXPrO0xsBrKeVjB0XqvzFmmg+cf9FcPYrj4tuG6BgyKYrCWMSQkIBCY1aaDN4BGSBj8FLvWkSBjLRpP5TpUhgC9mhM5+0P1xy11qxLh+D5xMakKVurS/SKZjr6/CE2cQEsJ0kkIXy4EW8WH552xpY+vA2eM6RS4r+gR1kpcve0vhLrgsgsmheC37v0L2qa/MxeBhkJSoPXsDPS+vB/VnSxECDRClopwn086xg8FoYMtu+BssHr7wV7STxvrSRqStMNBXgiJ13+8MAUGwOHH3AQuscHowAfdXwWUKFJiOKQ48HWSHpTejwSsPoG30riIRniDNd74OHUpnmlfVKrAzKVQ9srD5L95cy66E0CQCy3zEdkwfWj0OnlVQ590wZ+7rq5ind+KIlfQweSsEcmnD7WI76HknKvFIAzzr7UqtfxgevOKa1mSeWf5CV1sTtEy2LCq5vPEIRHSUGr1K6I+dNlmsPg2c/B/wcgOLZrWsmNtk3Cs5H+/ngY1OUoXMzyzS006TYiBtwzUKf/bFuXwTPWmu45sFNcMoRezdmJkOCVfmSgVPxPSeOiab2jx8VKfem9V1rSDx+nTj/XPbQfJ/sZ6B14SddML52/dRUOtcaUrKhxJ8jOYgV1oMAT77OuS5DBm8ou50cse/CaLPH6973fMbaGaHsUSzAG0RmksGLeTF2iElhSk00acJoxqLJQVYSmGAKGIKdkKLoE8SdNrj0M3gvPHZ/b12SD95Bi/0RJNH/DCDCRJMBOh5uv2KIoEx0XiY+lySVMsgzuXX1VuiRROfcxKnr5Ghzd1OxvC/lBQDAnuiDl2nHiZ5LpgMMnh48D56qQFfxGdnSNDFBV5B9Q6FsRQyDN9Uv7keSyC/2VKlaBq9qj5toEtbOlyZhpJVaCE4yIfIpa9t2GJNF3z0KgW8ulS9LxM7QFfeurz5LJrgSA4fCmXMZ4JmT/uwb18MzPvZrwQfP36YkCWE3aFE0/fKd/4PrV8HR77/A9A1BOo2i6RljrkwX12p8gaiJpm/MpLqtAC8sxyLKRLcPWutgyHfpkivTSuHH9/70djjyvec75SUQqQP14NClSX0ewMYMnmSiyVmfsgN05mW6CF7VTgcHeBhYZM2WHfCx8+9yfj//9kfgtV+9Br5//appm2VS4YGEgkneA9YWVjktf647j7dtpzwpx13Z9dYFWaGbcPzS6HMYyqfHo2hyE2I8NyZ/8FyUIYO3G8pF//c0WLMlHH1sKIWEqHmbwYtng5pK0yiaYh0NgrtQgGLlEyPHY9IkLJ7fdpSZTiuBqSwXGbxE2InrtJIKDNA0APQcLjTcvE+KKJr28kfz/3GhEVHr7oZjoskAXy8v0ySo4uW7fON2OOeie0CSJuCejo8dUlpm8FAKP4XiswI/Y2wxeLVpEvy5/Zr4n3LBHVv8jL6v1AcP2VEUCnANwPPPDcrgSfM6UQCL5tkRcjmIRtlebnBkTGHItK52mVFvtNIk1OyU+4Zw03YTNMZ3j0Lms1ymqj42U24lAP/en94B+y4agbNfeZyjXPM5Ko271OfxGgav7jqrdUzbSmiS0Ah9bh1f/e0ysd0QQwRg0hTwPlAGL9fufCn6QdoTbocVWCkzdVHp5xqmsjyck00Ysso3SWj3f65d6ZbXYQZPDCKDir7Qb19/6mTFxgn4yY2rTR48en/Y82EYPDP3slxDmhaWAoP64GFy75tXboGbV25xft+yo3hmb1m5BY7a31gG+F7VsexlZfIaEbnbx+C5+gIBZrmGh7fugPHJPjyRbXjx59tNSeF/TqRniR5XyqRJSJJivrjBlWgaEP81ac1/t58brGdoojmUR40cfcAiOPqARfUFhxJ8sOlPsUFWBurDDAZZiYltQZXfgsUozu1YvnkUZMlRNJcscpnPkVYC412j+FI5Yt+FcP86O8LW4vlt2DgxZaUBoOBSGht8MXDTSCodwUSTf3/raUfAHvPb8ImL7rEAXt1izxm8DgN8aKKpVOFDxq+ZShPzXJronL6wufIsvTjxJedLbA/QzAcPy0mSaw35gEFW0pJZASiuF+9xYaJZ3D8N2gLhNIomjlE7YAZmfPDkMCzS+Ex4ggmNTfYtXy4aTbMyoyv/ow9eYXZao9h7Or+VMHg+1i3L46NoInBtGhRHUobRXPrsVx7n+sw0ZPBQXIBn/17rg0c2vmj1ioD7GCzMWViA+IAkaaKs9BOaJDqXmA7+GcVm8LS33EQ3q2HwJPDl/81Xh+yD56+H9rmWwYvEWm/6+nWwfON2OPmwvYs2SL08GJePdU5Vsc6EGLzXfOVq0BrgB299pnVca+1YsXDZnV5PmAAAIABJREFUZ0Hxntww3m2c/iEkONZpxDz2Rzz2f8+0hmeefSkAACz/+MuscjzIV+i5xI+42homm7ddHBhtpxWDh88p7z3d3PIFccE6OeDDPhfnFt+HJppDGcpjUEIMymPVRJMyUAlRqGmgkZAP3qXvPB3u/chLYJ8FHaduBMJ8d/fph+0lMmiLR9tWIA363yc9oij7RPLBG2WMnlKqcpBvAvBarH9SkBVdmmjy39y64ucV+lokSlkvbB7hkX+nfghoNirJgjJ9RowPHoDfzy7LBzfRpKyaUgra5XxKE1XNzzy3ld0n7reInF/8x/yDKPT5LfLd2QFdrD6UlSwg88c3HNsmexajSpkZyuYBFONV5PCz68gFpdTXHh1XXz5IDmZC0hWYjxipy0XpmGg6DJ57jqT08xyGTX3w8JmhDBqAvZEQA2oks7JYEM2fN63BmRsA9VEVKSNlfPDcchPdfo0PnpE817Bi44QYJTQkuZYDaYWDrEDZhv2eOuys8+Cff36HWI8kNyzfBEvXFKks1o11rfJ0XXR88HwmmknB4IWCrFzz4Ca4dpmbFmIq80dIRsH7v2FiKioVhgYM7x+uuMqDRyLF+iSWwaNf73hoq7c+bgqrtbZcIii29vna+YDZ/E4Kkz3jg5coF6CGTKUdBs/6zS6TPcoZvCHAG8pQAqICT4htvjh7C8BMmGg2SbDeZtEyEchSM1Q70Intg9dOE+i0EtjXw+ABuIrHd//yFBHILh7tOACPAutQmoSQn1UnTayoiwDgRNVMlImWtwc10ay5Hfz3dosDPBNkpW7eNAH3LTI+FoPH/MPWj3Wt7zSxsgr44CHYLHzw5Il0yhF7V4nSfQpRrs0L+JQj9g5eExelgARZMSA+UVAxeAAGXJ58+N7wnb84uTqO87SdJtY10LlQ+eApef7gVD9oL7/PJkrB4KnKtI2azvUYeOr183LO2o3Sfk5V58tjS8v6WLcsj1fUMedhP8ut/Jh1UmfOxtvnAI/+3O1nsHV7T2RtOFDRPJhDDdVT+SexNk2AnbixqvK7BXx/fBIEeBaT61daAWxGyueDB1CwzbEM3ucvvR9O/8Tl8MD6iaLewAukz9hLyUTTBLEQAB65Zt7tb1+9gtXj7Qa86stXw0s/91sAMMG8kMUP5Qe96M4i+XzlC5YXGyFpUqzTMT54vEzMHMB7tWGsy6JLyudunJiCw99zPnzzd8uD9VYMXkSAtVBKG7tO851GguXSZwAryzUce+Ae8I4XHFX8Tuaqm6Yj3DYCvLxao91gSFb9Tj2kLW3/zn2cK9D/6MR3Q4A3lKGEJGiiGVluNvsQK6j8x7xw2lZ+MaPQUpaD++BRUIAflwjBaagpK2UKfQm9X3vyIZXJqPQOkrAvTZjL2bTqvEQ5DCMHfIlSlcnbHvPjo2hyhY375PVZmoRgXeW5V7zrufCztz0bznqJP9mqlSbBAgW20nHJXeus7zSCYKJks9dL33l6pQyHfLhOP2q/CrSd9onLxDJ5btIkvO25R3qvRxK64ZAoYqKZqCrIitbGR/A5Ry6xgiTRFAtUCaD3CE00fWOBdbznpU8CAIBjAubu2yZ7kCRynrRuxgBelkM7daN70qFGJdqn4PYtBdbDoOp4HzyAAnz1cw17BHxUudQpw1ypm2SsMu3fm75+HTztwxeLSiif200ZPGOGqS2GIyHPZsymmORbVvj31J/sADyS9LzXz+EDP7sDVmycqA1wIfvgCQCv23c2faz2ySlXP7gBAEz+Q1xXLrrzEVhFcsQB2Pci1wXLz+cB922yfzN9bpIvLkaQTaRziD8DvymDBKGpIJZNy2i9MYGG+JjEWToUZTaMdy02yedDi5shP7xhdbBe7oMXmsiZ53nlp9CvCODr2PY8h9JfTsFe5Ts3dB9wjvny4O1Zum1gaiClXHbSBpjF/+9dtxIOO+s8y1/aCbKS2+c82tMkDAHeUIYSkJD+TR/6mWDZfDIjDF6DICuWj5tSFaLhiaVRFFOEUQGvA3gWI6hctuqrbzwJXnHCwfCCJ+/vnEvb5oJKRaqUY+L4rhcdDU9+3B4AAPCiYw+Af/mDJ5P+2GUTVSjoADzISvh+8N87LDhNL89BgwnzHxIck0P3GYXjD1kMf336E+Cbb366WBbvW6KU9QINKXMAUAZZ8TN4i0fbcMS+C01OtDz3Ki6tRNXm7sMAI0qFzWgloaxaosxmRKqUxQ5j/zhDWvlxMgaPPmNT/dzKscQF5/dzj94Pln/8ZXD4En/KFcPguQCAB37o5UVIdt4kBQkhxb2oKxymHwCZiYYAL9MWi11/TjMFfdWmHd7fr3mwMH2T2CgXQMjKok9w+hVgzBxXDU00zf21xz8GR0uAHutbs3USvnPNCviH79/ijayKIvvgue1NdLMgALdSM2j7GLbx1u/cCC8rWTIUysJiu5zFM8yk2y41X64bt6YAD6Pchhi8SpTd16TcKKQMpe/5WbZhwvqOZrMh9hvbweTd/PigwvPghRm8cB0oVqqC8t5KbgTcVDnP7Xe8aKKJOFSog34/5fB9YMv2HoxN9iuLDn47epnLEH76V/cCQMGAVtcDNjjEz4+VPHhDgDeUoQQkNsjK7DJ406+j1WA3us2CqWD7bW8UTVsRxp+WLHR98Ciom0d8+pRSDpBFQPexVxwHV531vMoHjEoo2h4GMaHy1tOOgPP/4dTi3ETBm599uNMe6VTVJgWrdW5x/KVIQUanlVQMHkA9eJfa8oEiBNkKGFPUD6eksIKsKLdNfCmj6W4WMNFMEgV11so/v2UNLN+4vcov1USUMvdcKVUFsCmiaJogK6iM8fuvqrlsm7FSJWWqbxg8qXd8+PcSfE2t8sSESDRNI2xNJ1XO807Beq+Owcvcsk6ZPD7ICkAxf/p5DnvMiwd4tQxejbWbHDlUAHgMSLqmWnUAD2+mtsBgmqhGm2JYhAd3iFHSJUBfd/0SwLDMcz0sCEABuoIAz/MZwAbM25j/ow3winKTU9zfFxVnt33sKvpshaQp9ukKJpp194au0a00qZi2f7/kPjj8Pedb5y8q3xOPbJu06sD5FwJ41r0gXaqbu6ExGu/24YuXPwAANFWLv3wopYrdpvmMoFlawzlDl+vClxHXTvo7b9oXRROv9/Sj962OoXuIG0WT1o8gza4fP3OTzeK43c9hkJWhDOUxKCHcZgO82ezD9CtPIxZ5XhaAB7UgZaxUCrbSiwwWhtWnMuJh8ABcczgEFp1W4s1RJ40NvojTRDl55EKAgkfdTBTAR1/+FPj4K4+D4w7a07RZw+Bx5ZkCspFWGY1NQ8lgheuSdkd904H64Nk+TP48ewBgRTmTUgPgfcGx+8ENq+Bt/3OTtw/c55DLXQ9vg1/cugbagYidPqFBfygDmiR2moReBfD4PVXVcapAUBBejFdRVlKyeJ+lYEIoB+45z8pLKAUfsUw0W4kzp2kX0AzOFzBBMtPjkucasrx+7uHz0EUTzfkzY6Lpi+K5/x4jVhnnPOFyeDuhfFuSUHaDFk2UWVtiIjbi9XDfn5j1ls8nLfQ7UXIwCCoWg4d534SC/UwHo6KGgGQIfFG/PizCGbwKxAXSPMSMW9NE52iiGcOQ4d2orABKBg9B0LevXg4ALpgr6i//57p8zooDoRQ8UkTJ4vhggagAAM658G4477aHq/4DDJomwf5O1x28t5IfOQ34gz7e9N1iBbtB6xHTSNm2vHmz/x7zKkCNgbB4P+m68MyPXwq3rtpSvbdp1GoerMbMT1yvzWbxo1GGAG8oQwlISAGlO9qzaaI5E4L9a7JzD1AoziKLwQKdcJNNPJeLBfDaLqCy247on9A544MnMTj+++SaaCpYPNqB15x8KAvsEu4TV6psgJcWQVYA00vYbR5/yGL4w+MPDLblC4Ji8uBxBi8cya1Qbm32ggqOIb7If7V0rbeuNFFWkJ6QtFtuxMg6oT54aWJY1zQx45xrw2BwsIlXyaNo0mvuVgyeEgEZXxP2DgC8n7392ZAkytqV5uNrAJ5somn5U/bDUTSpouZNk6AL5bMuQuteC4r1rVuaaPLcfyEJATyMgMeFJo8PMZ1UXB88+/daHzyi/FLltTBVltsN+Rz1meIaxeCx71q763SilKiIUqGAAPshNd/L8hoTTflz0Yb/mrgPHoAL8EIMHjUrrcM2MQweHS/sB71sn48qDbICYIJh4ZhicKWVG42/HV5XxTJ98jI4/sMXV20sDAA8+pzSOdg0NQkAwBcvvx/uWzsG44Rdjdnc9adJYCCLDBneL8nPnTN4mS4tasqHyvZVlYGcL8hKooxveqJUmTfRLsuDHf37r++rntvPX3o/aZsFXfGaaDqX+KiQIcAbylACEgJ4VLFDpfO0o/b1Fd+lEhNJSxLcIQOwwZGTJoEFXUHZj0XS9PngAbjAJSaCpFQClZdEMPsMCTfR9J1a1y2XwTMnzO+kFYBQyn05fvgPj4V/f80JpC23Md810UTnFsDrZV5lBoCbaLo+eEkFqOrHEiPOxcggJpoFs2L6ZaJoGh+8I/dbUAGzNqufmm5aDB4phxHalJJBEsdFPoB3wqGLYb9F8yBlPnicKcb5MuUNsmL6wMO+c7EZPL/ZVa61NwARyoEla75xvAv9PA/mleQyFVBOd/QycR3ab9E80kf399nwwavYDbBNtWjQJ/48S0DXF0UzZkNNWk9cBs9mKQbJg4f3r5flNeuB+cxbweApklATTbxunirBmL5J7RqleiaCrEz2KOCEqu6qj14GT1m/46YVzrUD9ywB3ibjb4c14TxYtWkHbJvsV4AotDlCA5zQy2rqgzfZy+CcC++BV335auvFGJNqJzpNQmRfOEOntYaU6BK9LIet2wv/dv6sm0iWcl+o64VSxYYf7xef31pr8V1atKHZd/Mfx2UmchHvChkCvKEMJSCh55qbIC47+6XwLU8AjF0tJiJcs5dGil7MnvoAXP85yvmdWQZIQel4fPAA3EU0BiiEEp0nQpCVkHAF1sf2hVjA//fiY5zdZ9qHee0Uuj2SJoH1P8Zk0dd+FWQlcRm8kLKgNQ2yIpjKlt9jmDn0V4mRjgBmqEg/UX/PwgfPALzFox341p+fDF9+/YkkyApLUUGO+3zwuv0csvL+SExHLINXRTUlDF6utbWRoAjbWuTBc33wpFvn9cGLMdHUBfCoe76ecmBhlrx840TBPDZIBRPKc7hjKhPnI10PJGAkncOvsXEevMrPzl4bqYkmb0Pyp6oUQsZixSQ7l2wkeJtK2eyOCPDIOXjdqzcblgnnXS/TwSA4NlNolwsyeH30DzUbRj4GTxoXCkrrXlMxAA8jIPvOq5sbVRTNctMKx3e/0pR45SaXwXMZ5GJMQiaalr9Yg/5xqcZ8KrPm1HQYvFCaBF8ZAPv51booQzdc/99Pbisi4+Y0RQ+2IddrbUISa5VEufOUs5+5lt8n2vHBszdGTB4899xHgwwB3lCGEpCQMs/zpuFu0lyUJj54VCyFmhynyqFS9jjRIfjgHxwL//mGE+GEQxcDAPfBs5cfH3NU1z8uqFR0WknVz+cfsx88+LGXButyGTwPwAvU8TdnPMHZPaT1jnZS6PZLhghcEBszfXysV0J2Na0omn1/1EsA29FcYj2rPIQRCn4nTRzWzCetNOyDJwHKVBnVJVGGHcU+n37UvrB4tFMpYxxAU2bP54NHk+jiML7pmY+3opRS8QM8qPpgMXjsGUDlypsmofz9YJJ3z7dRQ03feh6Wpp+VJpo1QPyJ+y+Edqpg2YbthfmocF/fctoRYrTcOhNNaR2iaUqQ1aB57qQ5zI9pXYwX5nqsY0HoxhftEmWy+bVIAE+KkhqTb/ALrztBXFD4GnLtsk3w7d+tqL7LJpo2uM9yDdeR5NsjrTgGzzLRZMey3A88Kv9Q8jNn8Ew0TilNgrmOeh+84M8AIM9Bi8HzbIDg42el2yFpEvB20civ2B/eb6wjlGLEMle0zg3bqTrms2gmCtr2l2fWOxvHu06Sct89dcY5crOJDj1GRS0CcBV9uXnlFgAwKWns+myQhUJNNHFDDqMq86FyGDyQ361/972brc0obJGDzLmq19XJEOANZSi7gVSLfEO/7ZT44EkvDQDXrI8qqJ1WAi869gArYAoKzzvHgUWM+Z5UYrxb+B8sHGmZnb5EzrNHJdZEsw54cr2Cgoz57cJEE184nGGsC+AS6peVJsFi8DKvMgNQvDhRGWoLDBKOW51JH0Cx6RGb+uDx+ywI3mPptyQhzJhSxgeP3ROa7N46jsCPpUl44ZP3hyfsuwCOOWCRlUQX5Yyj94OTD9+7bNfuUy2DVwZrOfWcS0FrN9AQKma9vhbHDhWbfYm5sw83SGZ6Un25rr+f81opHLr3KCzbMF76Drp9Wzzahic9zs0D6DMPBShM5yQFno4LTlfKwoRYQZRca/jsJffC0z96Cawf6waBDIA9P2iflDL3mSu/kj8VTyMAUAazqUEiv3foXuITL7GvX7jM+A7Vm2jmcO/aMdg22YfjD7E313pZ7mV3AThTWNZHGI06Bo/2zWHwcuyfe76dB8/bPacNn9RFIvWBGrwfFcBTdpAVPI9uPhhgYNeJZcNmvXL6haYMHq7xWrPNWBZk5fc/fyX8/uevtM+NMPkG8DB4IqvH0oXkcl5R3JDhxwDM8TzXcNhZ58HHzr8bACQGTwXz4BV1avG9vW6sC3eThO04/sM0CUMZylAeNWJMkZq9NIpFufhMlzgrsTnwKJpCPeWCTJW4eTVRNGMWVanM2KQBeJWtfm1N8QxeXb/4GFOlfV47rRgiKTVEXGCZcL9okJVWouDB9RNwO9uxtfsLsGV7kRtoz/ltb/0S4LrgH06F6993ZvV9QacV7YP3xP0WWmCJm8hKAEQplgevPIez6egnw9mlHgF+VIHac34bfv3OM+DJj9uj8JHM7V3bhfNalr8flTqA10oV5Lmudvw5g4f3airLRVYTFSj67MT54PnLZNoN9sKl3VJw+JIFsGLjdujnuRh10+fnWuuDJyj4dMMHlbst2wnAq0m9AFDM5YvLIEDrx7q1DBpNhUCL0jQJHKyGTTQJyNK6dkON5nWkUh8i3z3Gc7xtKvN9HbL3KAAAjJTj28t0cCwlk7Uqf532ByeRAgC5PngGKLrtmrbqgHEM9qkz8/U9Qzx6appgHjx7LKzzKwZPbk+aMz0GGIvy5vcQCJcE7wtnq3iaBExab/UzYM5NRSolmmjSzA+6qAfH0SqnteOPiWCN+iYDFIngAYproxu3Ctxxl3xzfasd7ZJhqu0Nm6GJ5lCGMpQ5K4OaaCaJbEbnS6UAIJtCYHEaWKUuimaUD55QBBfnBSOtinWI2YDjQV9859QGWWFvGxrJEYOs4C4rV5pjQK1PMZfALCrNb/z6dd76tNawdUcBivec33bqR2VbMpk8bJ8FFrPUhME7dO9RC9SPjjBGV7j/1sYCCbLCgdy7Xnw0fOUNJ8IpR+xjHUelqZ0k4m7+SDuFyV5hwkaVkYUjfoA30kpFBQ7HMVU2mKRAdqRtAF6WF4FP+O3FPtP56XuMqe+Jz0wSo2jW3adWksDCkRZsn8qgn8mAkAZhAgD45KufBgA1PnheE03XB2/bpAF4IVYQ5dd3ra36M5WFfU8BqImmzVxR4MX9eSQGT0qTEANUEuXxwatBhjKDZ0fRRMUY5xtuEPSzPDiWUrROrJuG/sfvKF2BwZt0fPCK/xKeoM9BfR48+Xd6ngw84hk8vOY0KdYZfJ76pJ9Vu1VgEJnBk9YHfK7twDxuJNRYqUw0tbbmVIzlhW+euuaTbhnpmJWOQusyaJVrRUPvNfaZs7x8HGhu3gSDrLA+cFP421ZvheUk6ikVun5hUzwX35DBY6KU+rpSap1S6g7P72copbYqpW4p//55tvoylKHs7jJ4FE1boUaxwJeyQYdkr46LN2cvrD6y82KiaPoW3k6aFD54Hr8p8ZxYHzxy+KkH7wlvfvZh1u9OmgTLRDOx8qxxs7eYnUIvgydcq5O8XZAs17Blx1TBeAo+YPiCFQEX6/CCkTRKoRjtpPCqEw+22hplJrvm3pljtDw10dyXRWsdaaXwwmMPcNqtomimdgoEfJHPayfQ7WWwdUcP9pxvzKoWzWtV81Uy9cWUAlQqppEEWSn6Zpto5tokt6ZpIKo+I/gk58X44Hkj45VKet19aqeF71E/K3w4JZ89urlzzAGL4PhDisAsqAxL82/HVIQPHrKaBCjGhI3/yHl3wdKHtwEAwNhkLyLROQF4pGhC1jQOlBeJDJ7Z8TfBIuqDrPh8e+quNcYHD4ED3gNcs4s0Cf76bWLKBiFZri1AQplA/Bwy0TSMoD/Recy4+eY/PU1iwOwomkUffDlWK5+vxE6TQNlM3q7jF1eZaAoMXuYyeNzMNiTcLBHbchi8ipH0j6lvI8Rnoknrr2NKKx885UZo5hsG2H/aFt8somsOWhg5ic4zDSNkvRojaSO40PXcmGiW9eTu9T6aZDYZvG8CwItryvxWa318+ffhWezLUIYyK0Jzls1lGZTBSz1BVigwocoQgC/6YfG/SaLzqNDEniILSjYI+xWzQEuJzsUmy8raqYJfvP058NqTD7V+DyU6H+20oEvC8LtKdgyD5+uX3T8AgM2l6WVIcg0WoPHlaZMAAT822m7VJjoHAPjca04oGFbSV25mif2gin+izPWliTEBW7LQn4uOSpUfj/ng4eeRVgqT/Qw2T/RgMQFti0baFdsqzYu9F7iBRrD/1LwL2zCfk6p9kwzYrgcVvRFqwsge42d87BL49K/utfPgedMkFOfXmWg+6XF7QCtRleLuM5lNyLzDdQHb5ps4ABhkxW2PmmyjMkoBRChwiyTbdvRrGbwWWRepAkyDS3E2bYEE8Ig/Fb2nMQxibGRCq70IH7w+uwepKqLO9nIdHEtaM1d0b1u9BU4957Lqd5rcXDbRtNsxvnz+a8b5GRJ6S65+YCPcv67wo+oJmzZS+wAGAP716UfYhRSeX/wvfPASw7gR3zCUKjAIa9MAPHcDCOe2tNFE2/GJD0xqbb/vENiHaotPdF7WSdaCOjY5Lxm8VAnvFmYavX2qD1fet966Hj5XFbEaUEqBAvcZmspyx0LIJxZj7bmmIYPHRGt9BQBsqi04lKE8iuWzf3I8LP/4y2al7m+++enwvpc+aUbqojvVTSRJZHDEg6pYUTSlepRRdrEoT5PAd/dimCDfwotKmDFbjAFOcQATDyNwc/wK2FuRsp3zrCArbs64KExLrpkCG7y3kk8Byj+eeZRTX641bN3eg8WjMsCjpkpc+BiNjqSiKSefxwgC6e3jijMGoKEgQSkF89tJ9XlLGYRjn0iA95fPORzOfNL+8KfPOFTcOZ/XTiofpb1JGpQFI6nXRBMAYB/BD48GWbHykQksdqZ1yZJFMnhMXVu7rQuf+/V9li+N3wevCHISMoFe+uEXVUFwuj3Dej75cXuwa7R9P/FZQKWMB1ICKFgdSfkeEUw0+8zkVPID9EkMg1clOgdbAU4DJpqjHfeaKh+83OQ5jDXRlErUmaNKl2Xde2qi2TLPSytV0IuIqotSAbyyrgnmUzclAjw/g1eZaAbAF5r0cVm5cTs842OXwOrN263fX/vVa+DMT18B/SyHf/3l0uq4GKlT2NTxWTnQda+VqOp7Vv2XgIFchxRFkwNGeox/jhF7TIm1jWdzlwKbmHQe9Bz6LijYbw3n3roGlm0ocgPSZyYvrRO4Kwe2S/v1/35yO3zr6hVWnyQGz/hhK/EZ6mfxeTvpOH/32pVO6gRs59Eou9oH75lKqVuVUhcopY7dxX0ZylAay2yGzz3j6P3gr047or5ghNBgAk3PM7tl5DgNqqJcwMcFD2HSWIAIBi/KRFM+jj4PqMTG3Cb+PvDmwQNk8BLrPwpXXuiLZn47hX65g67ABYdRPnikzJdff2JlIoovxFAdHFQDFC9oyuC5DJKfweMy2kkd4PCjv34mvOL3DrKOodkqVRQ4GMB66PE0URUQTJSqoixSc8qQ7LNwBL72ppNg8WhHZPBoWzTPZas0+QWQgf+bnnWYcwyLtVI7qmlHYLHzHMpIc0KQlcoHjwZZka/PBq1+H7ws1yU7L9czvxyHVqJgsp9Vn//3b58FN7zfBNWhQVbSxHzGPGs+Bk8ysaPrgbR738t0o7yW2yZ70YnOQdtrI70PXNGWrqkKypDpimmNNdGUxqIOmNZFM6QmmpR5bpf5H4MMHqmamp5KYplolvNEk6q5Dx4P2mJdU8VAyXnwfnDDSli7rQs/vekhcf5ffs96+O61K6vvsT54fP02Jo3Fd54HrzJX1W5/fWkSOq0EPvsnx1u/GRNNalYdZiBDQsvTJcTnntGELUTJq3eB+877u+/dDC/59yuK+hgrqbUxdbXq06YfSgHct9ZEtORBVlC4iaZSygXW5DmsEzoO5966Bq5+YKPA4EVVNedkVwK8mwDg8VrrpwHA5wHgZ76CSqm3KKVuUErdsH79+p3WwaEM5bEiuEANZqKJ3+iuIDXRVHYUzcBiSFkrh8Fjq1FUmoQ6gJeY3ev6uuwyvjMcBo+9tPiuPfVdwmue7GWFiaZHuQgJHZd2mlQ+Qfieou9ezpxJjEquNWzZYRg8nw+eFPWTy7xWKgaO4aweMni0LQ4gEchShTpRhkFJFBCAF8fgUaH3qQJ4pC0cj6rPFYPn1nX6UfvCKxmI5WkSUDrCfOjneRXZkk8BNCulAMhmWdzrAPD7cRUmcNobQAmAmsEmVT2tJIF57dRiKwsFy5wTw+D95t71cOtqN6qrFWRFAHj9LI/aZEDZtqPfING5DSoUMSfjYIhvTAHYwMUweP7gFSiJksF6HXsjBrZgCjuChU6Kz4uCdlqY3PYy7VVapXeED6haDJ7kg+eNoim0QX6T+kCtMCRQzMdauvd0WDMPwMP5bBKdgxVkpYrBufWGAAAgAElEQVSimWt47icvh7d+50Zvv6oojImCFz/F9gmuAJ6QoL74XOeDx66NAjxyvGK8AuMTy+BhVyVTS4AiBQqAWbOKuqFk8Nx3C2drtXB/+POHfnf02pxE53kDBo9dexHl1z42zIPXULTW27TW4+Xn8wGgrZRa4in7Fa31SVrrk/bdd9+d2s+hDOWxIHxXMlaK09zFzTFnrGXwSoWRmHFx5Y+bUcYAPN4Wgh1keSofvNqa3Lq8LwiFvxcf+G4m1z0o4EGwMtnLIVFu2OiY9wgto5QZW3zJ0pfRIXvPhz8j7JLE4BkfvEJxd17eRAGpuydJohyFKU2Uk/6hCqBCjvPxl3zwlFIw2jEM3suOK5Smw5csCPZLEkmZoLu+ezGzyxEBlIaEmiX7TDTx2tBfi0aIM31DHzwyWKTvPY+C6BP0ifGF6KdCNy8MG26OUV+1VBl2M+SDd/k98iatFWRFo3Jnrmcq02KgH5+MTfbio2iCDUzofeAKpmTSl2kNG8e78NCWHdV9QtO0kHDzXZRa5V4CYdZ8NoFUKPPcLoPm9LK8Ymndut3PvRiAV36mQGu7Lw9eoP8YlMMnGmRQLJn/OW0IrD1fr7S2f0+TpDTRtNnMTGtYtmGiSstBr6Fqo9occTfHeoKJpmRVECu0HucdIfQtJicgF8PgyddS1UeeGbreSBGaDXNrm55Xc6/vgi3cmEF/PCw72cvKKLE62gePB7NJBEZwyOA1FKXUAap8UyilTi77snFX9WcoQ3ksS+Vr0pDBo3nHLMdtsuL5cmFZ7Zc/F+Yurm8VQJG3ikocg2eXwVD7C5kPXswCzX0A2y35JFQkkIWq8wuiCgQCiIrBGwDgWQFtwIy9CTdt1WjnHpQYvLzwwTMmmjKDBxBnpslZyVQpWDjSgj94mglIJPmz8WvH6xphJprI4CkF8FenHgF3/+uLvbnoYoX64KHs5WHwJNAiCc3VxPOjoeD96OfGbJLPadSVfHnw7MASMjCgVWaBtnzXAOBuZADYzz7dsEBlP3asAGzghEqfxeDlDRm8yX5tYJYqAIXmUTTN2HDll19TOy0UwhM/cgncsmqL5YNXt6FWBFlxj9ebaLrHMqaw80imiSqut5dp6Ge5uBYAMN+sAOMGANDtGwAn+uCVDN5/X7MCLrj9YWPyGWAoERCERGb4bJEZPBdA8fvJrzlVZTRZZO7K3x/avKO2X4YFdCNIGgaPstT0ua5jcVlb5Hr5hmuuAX5680MiwKX9rBMcA2610mVA3skHqYt1kI8BBfOcRTcmmnbdaJaJ16bAjPsxH7gQ3vj16yxf2NhrqkQ9doKsuJ6fMyRKqe8BwBkAsEQptRoA/gUA2gAAWusvA8CrAOBvlFJ9ANgBAK/RTbXPoQxlKFFiTDSbn4tPJV3ieNTMJPEr6vTcJDFKIDd1WrXZzlMTSpPwxP0Wwni377zU8Rw3iqa/rrNfeRwsWTjigEBfnjDFfpfCx/vqQQCxo5fBSDuFdAATTQ6KuPmt/XK3AZJkXjaV5TCV5bCgY48ZStYQ4LVZmSJQj4LPv/YEOPfWNUWZ1ET2Q6ERT7U2SoTPRLMISa+8imoTwbGjkRwXj9qgEfv8+H1Go+rES2slyvYlI6wHzgfK4DmKasWGyVE0KYsSsxOPwCNJ6jc+WhbAcwtTE82E+uAFTDR9QhWyKgcWG7c9F43AhvH6yLAAANt29Kwoj5LYic6pgmx+c0w0GTMwr5VairMVRbNGpZHMywAKBXnRSAvGunJ4dwngYD3tVMG6sS7c80jhz9QhzDOaGvYyLd6bTprYUTTL/z6AR+detxwn2rVNE114y7dvqFiuc/7oqd7+o2R5eCOSKvRU+P5DrA8eZ2Q5g5ckGAnXNqlcxzYkAdz36z/96FYAKDZHuO8uZwR5/6blg0eO4zx+aMsOOP+Oh+W2IgO64Ljzd/Mke85sE01k8IR7pDXxZWT588o6phiDRyMN43OKgV4AAH73QMETxaQIou3Q+l0GbwjwLNFav7bm9y8AwBdmq/2hDGUoRnCBit2ps6VkhsgaF0p0HjLRRCUDwFWUVrMd0VCahF+943QAADjvtoet49jOwpGCfanMyrw1QZXm4KEtdvs+gFcxeJ4omlzoWKGSvmMqg71GBTBU1n31e55XRS/0tQ9gm2hWPniWeY6yWCmeigDAhDpHRtIXRROgHsxKZSQmtgr8YLGRZXmloK91FXKfs8Voormd+fdMR6pk4mSsFs9vw5df/3swv2xvy44CWBy2T5w5KA0+sqMnm0PhfOijEiT4xcl58MzvdaHhAQAWdgxYQB+nduLmPPRdA4C7aw9QMnjVs21YPhzPJgCP1l/lwGKMxsF7zYcVnoTFXLZN9qzzJbFNNM3xwt/UtEuFb5KMtFNLMW2nCSgVb6IpLckTU33Ye2GnEcCrAnqkCVy3bBNct6wIYj5C2PJOBfBycS1opbbJaN2ee10UzZtWbhH77eQ+Y6xhnYmm1C1f9FmrfQtAleseW6+wj/i/lSTluACs2rQd1o/bwG5BJ62ii/L7gvnXpDWQBuaR+lxnpuvUR9rGSJQA9vuAArNBGLyK1WRrAQ+mY5toQpkmQTlWADSKZq55bODid77BkpI1Mil9lnOtK39slFjrAT5PEmHT5VGK73Z5FM2hDGUoO0FwgZouSX7MAYsAgJsJ2i8OaS2kyrthZmwF4ykH2SHY/3977x1myVHe+3+r+5wzaWc255y0WmmlXWl3FVY5J0BEGZGEJILAP8AGbAuwLtE2xte+NsbG9sUiG9sIfC/X4dogsMEYEAKuSRIggwChCEorbZiZc/r3R3d1v1Vd1d3nzDkzc2a+n+fZnZkOVdXd1dX11puq+eBlv1+4c0V6nUMNLawE/kZZrF00hK++KYsQ6DPx0HVoHzyfIKgxBbwsyEqg8teoy169cAibPH5l8hQpXOvJi5zoBMrUSrk+etq8Rl+H/TGT378qz8QWBFyaWFfAEvnRBjLBzl5M0Bq8w+P+5LXtsHbREK45sBGAea9qYYBLd63GOcfEft93JyHAfRo8W/eW+qYFZhRNOWFJffCaSZoEh7+iK5y7nEiON/0avIuPW4nXXnQM3nhFFmxHr5q7InbayL7tevbaB0bvtzV47Zho1sMAd7/zCiwerjsDLIw3W6npdRUOjTdzpmP2daS/RxGkY6NMOG+nLLDHhYFaAHmINsfzBQuR+Hzwnjw66Uy9UYQv7L82NU8Tdid9zfVe1gJT4Cz7XMj+5sqDZyP97CRSiPYGWRHNLdJgZuXkBSTbjBXIm9jrQ/R+HWQFAM561+dw14NPGMfLBS1X4m7AvQjoSpMg21waaMf627eYIL8HMtqwrS1cv2QIzzp5Xe78UfHOpX6L1iBlL0ZOWs+zmVhb2J/KVpR9t1qxE555TVGUS5MgxxyVmFK3ojhVjKSqBs9+v1159Srl5J2FUMAjZB6gJ5I7rTxWVchMNBU++pJT8cHrTjEmOcoyvXCZQ0ozrtRE09LgvfcFe/HpXz07/buKOWCmGQTed83+tJ40hYHwD6qC/AB6TTSTsvSHvUzokRMpLdQemWg58+C17YOnsuijLhNNpcz77PIX1JNxva/INLbKM8kljHeco6/bEN4sf0m9xxBYA2A4mXDYObk65fO/fh42Jlo5bc5qR9AEgL0bFwMATli3qFK5utmhUqYZpZH0XETRbLmFrtgU1byvcjJrTI6tSWE9DPDqC7ZjVOTharUiHB5vYqgelq572BFbXftlRDs78uTxa8YqjzmN9H0KRB68/Op9VQ6NN3OTQ8AUOrVGIZ5oZsdIE007TLs9cQwC83nUw9gcT6ejKCJQboFoohlh6YIB73luASffPiW0qipp23izhYnJlkezHhjtKVsOlO3QlgBFERlTDZ7V/ieFprI0yErkvme2KZ/TB88SauT9sdso084UjXvSD7EVuTVvhRq8lnuBpsx/1MYnEMqq5X0/5bdvxR33PZ62ZenIAA5sXZo7/3UXH4MbztkKQJhoJoXqQD06lYqua6LVSvuh9q2Tmre0PeJZN/PyHZoODZ6ZB0/XGeFnj5qafZc7ggu7nyjlMtusVNSsgwIeIfOAscE6Pn7D6fjT55/c9rl6qFMqziN2zjHLLSFCWXnwXKVkk3c9URy0BuCxwTq2rxwV5VQR8My/9beybglfVcdnc0LrPksfUjXpsmyjFrbG0zx4HfjgWUFWMj8is316v7zPoSNQhl59te+ZC19y7OvP3Ixfu2RHfIzH7FTiCrKiT9N9KQ1QYGn5tK/gIY/5WrvI69WTNVdUzlecuw23vekCrF005CzHNjDS1xFaPnjbVy5Ifx8QPnitxAfPptlqIVTKCPojayrywXN1p2YrwsEjkxgdLM8dKJ+lM9G9kjmpMt8Y3aZzd6zAP73mrGq+m2n/ywRV28SySrjym55yHC7cuQJPHp3E0cl8YBZTKxn/jCwfPDmm2UKi/Q4EShkTZx2BU0YI9OHLgwegUIPnst7TbZDXVw8CYzJcDwJMNiNMtPwCHoxIhsXtNzTJlomm65mn6SSs5/pzYfZ4eKKJv/vGPblzZWlVIo9W8cFzacxtH7zYRNM/VT4itFetKHIKWq57PeFImC5/L/MftaUhn7ZY1m3fko9+5cdpvVIDb5wfBli2IO6LmVYzEfAaWdAwwDTRXr1wEABw76OHUx+8XBRNsQgS5xOMcvuL8+DFwt5dDz6B6z5wO4BskbZTH7zYL9Y8pl998CjgETJP2L9pSaVJHQB85rVn44+vPgmA0OBJwcEa70ytkl+DJz8irpD9vjJ9SFMNIPvIa+FLf5irjs+yyrrnA6HrKjPN/MQrDuBNl+80k3kLYcu1MtxukJU4yEUiEKVBAUwNnvSDck269Ac0S11QpMFzX/NNTzkOv3zeNgDuj7iNNvFx+W7q+l2J2wOVBVnplgZPsnvdIuzftBi//YwTcvvCQGHF6GDlsqTpotayXX7CKrw8WQ0Hsv6gzSZrHh886bsK+KNo2pPcbIEjK7MZRTh4ZMLQ6vkoW/CQ5lK6L8qgMjJwThlSg5ylSbCuRykcv6ZYI3j9mZuxcelIbKI52UxTpmikBk8veJgGmlIzkBeabQ13qMygDIFKrqFCHjzAbwa5dIFfwGu2IrzwL7+Cf/v+Q8Y2wPLVRGT0w3pNpUFWXIs12tcsO78Y2d1sE037vss22vdFRlD+i8//ED99OB+hMmtTXhAA8hosp4Anff1akVOrZCd310FWqhBFkVNz6DpfC7kTHhNrW7ApwxdgSV6fveCQmYm2/AKeww1A30atwdPCqP48TLYibFu+AIP1AHfef9BrnRAnQc9MNF0aPHuBRQqKWtMu/efHhuJ+V9lE075vUf6b1afyHQU8QkiebStG07D2etiVk0RbCCizUde7AyXTJLhNKDYsGTbOqVKuRq66AsDyZJJUNSCZFE59Pni6Trn/DZcdi0+84oBx3MkbFuGlZ28xPmqGWarKBySp8h0x/B0VsCZZKV23eCjdJq9HCtKuD7jWWukPYpGJZiepK1xajNQvyNLOyTomHQKrDLJiJ1HuBiMDNXz8hgMdmTLb6HZLDd7udYuwXJjepWkSmvFKtjPISjNvSiZvaZEPnmvBYLIZ4Ymjk1gwUCudwJsavPz7ILV2oXG9pnCejw2aRy+oBIFK+4zLRPNjLzsNu9ctLCwrDnwxiSMTWXRYjTTDzhKd5/Pg6X48kfMBUtbf+STjQaAqBVmJ63Yfs3jYL+A9fmQCX/jBz/GKj8RJtm+94wF84D/uBgA0rGirurWx8B0HWYk1NvnnWQ8DPPzkOK549xcAlPvgZbk384nOXf6SPh88O3BJGU4TTauvlAVZiTV4gXdBSpoj+iwXXO2y+6wuw+YlH7odf/Dp73s1eC7z4sK6vT54wEeuPzVXPiCeWSt+F9xa+vxix8EjcUATrdnTvq76fZpsxiaaO1aO4s77H0cUZT6gEhlFs9XK97dWKx/kyEzdpNJ/6f6kx1f1/7XviU7rYN6D/pTwKOARQtrGHu7kKr77+MykQvvF+RKRfvyG0/GX1+yrZI5lH2I7zq8Yi4Wfh5+sFlpd4k+TkNfgvfycramPVtY2PeHNtsnJZaCqmTPm6rcEuEt3rcL7X7wf156x2Whf/LspSLtWktsy0ezAGcG9op2vKxWILI2kaf6rcMrmJTh3x3L8t6ce13ZbJE8Tefl6gRR8tIAXKGUlOrfSJKh88nFtPtWQJprilkoBxDZ9yyZC2bYnj06iFQELBms5TchLz9qMH/zWZenfUgiwI77q69HPLfVNNa43d4qXhuh/rkTn+jrGBus4bk2xgDc8UEMUAU8cncxpkqTmUkcDjCIzB5cUXPM+QGZdYaBMP6pmlAbWKQuyAvgFPJcGTKM1JgrAJ752D67/4O1p8I+GmESbudHid1xHn93sCBak3+/v3Pt4Yds0uv8M1sJUINHbXBpiO4rmL544iocOHq2c+sIuR1JJg9cy98uE9ppI7Afi8ccOKlLUrqLxzubdt/7A6OPyGsoEPLuWIg2e9im279uE0ODVQnfQpSDI3nF9Tx49rAW8eLFK90e5OFcLA+xYNYrv3f9EEtQpv3gYRVYePOuqmpHpgycXivVPBVOo1uai1YOs5PsN8+ARQuYF6Vjn0LZo9MDtE8rkZr2K5wuhvnJsECvHqpnC6fp08WmUr2SyqMt54PEjlcqT+Hzs9LX4TDh9bQSAFaMDWDLSwMNPjkPBEWSlQpF2BFOlFM47dkW6zdaKlWnwciaaBR+zqivZkmWOYBH63toRP+W2NGiMuCfx9YT4wLWntN0Omz967h784S/tmXI5PqRGazw1YTLvX5bovJUmA3ZppQNlhhiXgpmtwQtUpuEIrfcDQBpO3DUBH6iFxsJFuQ+e0LwaJqmJAJJKmLlTc6RRU1UWddRloqnrdaHbOCy0dragNCYWWeT1yXuqVFaW9AM8sHVpqkHWBEoZKTu0oB4HWXG3U+IToYYdaQw0h9Ow/MDrklxrGntim/nmKgRhll5k5+oxvP6SHTjzdz+XHmsvalWNojlYD3Immq5ARXaEyr3v+AwA4IZztqIeKkMAzZE8+z/8zA8w5ui7tnmyS+C5+Ys/wulbl+Ki41bGQo1DYx5Fpk9m2xo8R712WgGJqcEzo8a2g09bbERatgW85Jk1I9N9wj7fHpP1M9JjuxaqZJCleqCweKSBx49MpGkS7PGv2TKjaOY0eFYUTXshWWsX9aUP1IK0LVWDrNjRVl3t6GBdc1ZADR4hpBCHfJcb8MqyEeiBOUKU5cFrI4S6j5z/hDbRTOpYlQh4DzmS0pbhj6Kp97uv9pTNS4y/TZPKzIcoCPKT5iorhXZ5NnYUTdMHryDISpoQuaDuiivZmm+95WIsH80LeEXt1rc9NdGUQmAXv1jKMeHoJlLwyUwWzWO08K1XwGuByt2byVYLSpn98a+/+lO86//eCcAUgo5ONo3jMh/VrDwt4LlM6Oz7YebBy998Jcy69LMJA5Vb0a9yl7NysjDlrgh6drtcZUghzL7OkYGaCDySTVzl/DgUk1rdhk+84gD+6qWn4bg1Y3jXs09Mj1WWgDfRaqVmpo9bublc+IQoW5CU6PpcERvzAl5m6lwPAxzWATHCAOsWm1q8fMqAYglPT76H6mGa6FwLGkscQWLsCJWanz9xFMsWDFSOkvr4kXyAJVvj1fQISG/51HfSdrp88ID4mWjNTizgVRt4oihymmgWWT5MNFtpf3SlnfAx2WrhtX/z//Cdex8DUCTgZe+F3bSJ9Jm1EhPN/Plh4PdH1WO7DjSTavASH8+heqzZjaJ8MDbd5mYq4LlMNM0gK1mkZaHBs9wq9NjTaR48l+9sFWui2QgFPEJIIa6PvD3ghdbAmz8h/tGKsmPbSYLsw67NNtFcORZ/gB7sQMDzmXjoa/T56H3oulPw9Zsuyh2v2bU2Ni+bmIxyQmSVz4gszjVvkNtiAU9M+B1N1j54dYfZpE27JppVg/oA2XVpITQ1kbJ88PoF3VSp0bLvrQ6yMi4EIleQFdtEEwD+9F//Kzk3ez8PjTeNfut63lKDZ7/Z9vOVmgvXsw8UcsJSmPh56f1AtSAFqUmzocGzTKVKxhndxhFDg2eOM8P1MJdjMbJ88FwmmvLZXbVvffp7GOTD/Otr0OPOSCPEWduXOdvsM4Ms0uD96OdPJG3Ln2uPSzJ9Si1UOJTkj9SmnJ957TlpsIycBs/bgpg3/t23AACDjTAXRdPlQ6jn0vak+uEnx7FkpFE49pR1IbtMn8miFEJ8WquW8A0rS5Ngn+eqt2jcarai9Jm1Y6L504cP45Pf+Ble9VffiM/1JEaX+edswWVCB71paQ2ez882WQixri31wZs0NXixyWdgfOPDIL+gJhdWXO+Bz0QztdxR5veuUWtfwLMF41bkMtGsVNSsgwIeIaQSLnM6TboyVyzfIYqidALvE5DawdZQ2CaaRbmkyvB91PVWn4ZvsB4aq9f2t31X4j/0/QcPdqTBczmUF+03ffDybdbfN5fZ5JnblhkRJTvxwauKK7gIkPfB6xekyaKe9Ol7q7W8WhhLTThV/om6TDQlcgL05NFJ473K7ldW6iOHYn+nBQN54dvuj7JO14KHHbI8Pkelz66dICuyDV4TTW0poMzj7d+HhdbONtEcboQ5H9AIpvYgnjiaGjxf35+YjHDn/QezvxPBoRXFpuELBmr4ztsuxSvP3eY8vxMB73Pfe8i7z35OuvhAxWOu1rZordS2FQuwNgnQZJsilplo6rJiTU3TOMct4EXGT81EEpSjKMBTGbbmzKfR0s8x1eCJ53pMksKkFUlrkHYEPHc+uqLzJ1tR+sykcOgTUG0iAJtu/Ad89Ms/ce6PF2Hi+m2hMTXR1FE0HfdficUOu02Lk+9cqsFL35kI9UBhUC42qfx9kP5u2jTW3u8y0dTjgK3Bk0JdVQHPXiRptVwmmv3z3ZHQB48QUgk5xNkDXhpowXeuOL4eKjTCfPSyTsj5K1k5mMJA4fxjV+DyE1a3XbbPx05fe1maBI19nbvWxiaajx6ayH3w2v2OuI63TTSliWRxjrv89XzkJadax/TmQ2evxALutA/dNNHsNTKKpkZPgD5w7X7c99iRVJsm/SDta5xsRmmADBdyApTT4Dk6iJ7QjA7WHAKU8v7tFPACU5DVPzOBFsZPiVJuAaJIwLOFSR0kQxtIujR4tonmUKOWHqd/2gEepOmp1pD63p17HzND+jdbLQRBXOZDB49iRfL++c73KF4w3KjhQ9edghfdfJv7AA92P5EaPJ9/pZ5Y5zV41YSMwXqIx5PIiro+tw+e1hjltSYyYI+LsrHRzpko+6Ddzzbd+A/xz6XDxrhz5Z61+L1//h4iZJq40LpvRcQavPwDLbquyVYLA7UAB5H3B9MsHWngF55AYTqA2G13P+zcLwU3mZQdkCaaSQRfl4mm9MGzfHz1YpIuV1/nZDOvwZPvlKYptGXNVr63xVE0XT542cKuLFGOUdXz4FlCb5sa2NlMH30uCSEzgSsPXk7AswZem0yDFwsS9TAfTKITbFNR3VYpnN384v149t51bZft0zCmPni1ahdg3xOdBgLozAfP1RazDLM820zGRxWNars+eFX4k+edjFtfe04uEEBmIpUd208fWpe/mP51uFHD1uUL0km2DMJi92mtwbNNNDVyAvSThw9ZPnjxT9dtWzBQy61eb7IiK0ohwLUiLlf3UxNbsQhQ5IPn09YEgYJuVs5EMx1nsjLk/dL9U/qvuTR4OuiFYaJpza11sXYQIs0LTtuAa07fiIOWP9hkU5poHsGKxEzc9+r5/NyGGyHOPmY5jl016j7Rgx1coikEHfls5ARYt8C+xgpBQAG4g6wUmWi2LG2NNmst0uCVBnyxFgP0e2H3W+nTZWutUl9xYbLrCu9f1EaXBq+43cJE06O1255oFl1I82AXUrA6OuG+R60I3vsfCCGqGUWmhUjyu+2DN9GKffByJpq577X0wcubRsYmmnLhRbcp/8wAs08XBVn5+A2np7/nTHsdPpR99NkxoAaPEAcHti7FfY+1H3lxvmBHe8yiaHqOT7ZHiE036rWgK47LviKqhrUuwh9kJdZADterDZ92U5RSuOKE1di8bMRRdntt9IW1tsv75CsP4OZ//5GRpsHGnsSsckQy7YWJ5hUnxtrVNMF58pE/f8cKfPqOB3DdmZvxt7ffEx/TR19aKYSk2zwastQHT+WFoYlWq7KJJgDLB89/v2QUzV/atx57Ny3GpbtMTXfoEQjS8oWAZUfTlNucQXWkKkDWqTLzuIlmrN04KkxYjboCZUwKUw3egF+DJ000jSTQ1uRST7p1VD77Xr7j6bHp8ge/9GNj+2QyuW1GER54/Cj2rF/kPF8ja5XayyITzSL8JprKWMQxo7LGP/Pav2p16mAa8Tl+DV7LEurS7S130Kl2sCfq+r2QZqlxXdlxobWgoquXOQxrgf/ds2m23Bq8snZrjbPLrPRXLtyO/3roycLzi5B+skctE02t9dTmqs4omkIwayYavFteeQCPHZ5IhS87iuZks4V6EJj+3yr/fJsi911soglrf2S02XbJaEXmN7OKBu+qfeuwXgQXsu+5S0Dvp++OhAIeIQ7+6qWnzXQTZg1ZovOMvIkmcsdIpAavUQu6EkFTtsP28emGKWHRZOP91+7Hjoor666Pw588/2QASCOgFR1bRNnhevfJGxbj5OctLjxWTu5uueF0bFyaF0B764MX/9Tf2+WjA0awGnlMP5CaLEqNlvXA9MRxXATysPuAnny5TIZPetu/5PNhOUyaXIsJcnX98hNX45xjlueOMTV4eYHDNWkz0nho7V7uTL8GTws5t97xAL7wg59j4VBdROXU5WYCXmRN2AFTg2cLSkONMAmPLyaKDvMw/T7olARV+/5ks4WBWg0trcEb1bqususAACAASURBVBo8j4AnKjYFvFpufxXsSJgtoQmX46I8Th9jj5tV8vgBwIAU8Cw/aKM8j49ZK4pQU0HheFZqomkLL5MtZwRMGWhkqFFzBnFqRaaJePU0CVEHGrwWBmqxttMW1hYN1/ErFx6DV3/sG22VKQmEBk8LYhrDRNNjIhtvR3qcgsJJG+JvyWfvfACAyIOn4uixrSjuSwNijJGBizRmFE2HiWZJHrwoMe3VyAUM1xzjwp0r8K5n78Y9jxxKt+UWBlwmtn303ZHQRJMQUkhmoplf6dSURdGUpi/XnrEZb79yV1fa5ht4q/pMdMoZ25Y587u5aCdwQLsfElfwCtMHL7//m2+5GBcdtzK3XU749m1a4kxv0CsfPMDsI4Db/6eXaQ26jYwKqbEVAbbpVBgEzmAzSrkTjT9yaAKPHjJD8UuTQX3KMStH8aU3nI+LxXOXk6EhT0Tb0BDwfEFW/OcEBRKebwElULH26/oP3p6rN4uimZUh3wHdP6Wm2hlkJUnqnC48IW8qqTUAhy3tRBkTiUD+5HgTRyZaWJJEGvSNA1KIksf4NBD7NvoXalxamDQnYmD6ksnfpZ+exGWu5mKoHqaLFLI+GzmXNrR5kV+DVBVbczYhomTKcuUljQ3WjP6baYYywSNU1YOsRFH14ChZu7MgK7Y2yTZJ7kSRFCjpg2flCmxmWtcwcC9ixGkSMg0ejPtlCo5hoFIBqR4GaZRg3Q77eyTNMl3555otW8Az70critrS4KVRXIUQbt/zCUf00n7V4FHAI4QU4lrE9SU6L1PhtaIIO1aN4uLjV3WlbV4TzR4KIu1S9HGwBbR2zVZd8w7XhEUyNljHqCMHWhWhuKoP3qqxQaxemDfx/OdfORu3CP8HiVyVjX/6j+kH9HOouQQemPsyDZ472IwdIMOFNrk0BDxR2OqFQ+mkJw7mku3zCXhGFE1nHry80F1zaCxdT03fitGBGm570wXZOYEyBAs7fYH8qYOsyHMBc3InTTSVihdn9PWbPldm+1IBb7xYwJNmaEA8aQ6UwsNPxikSFg3FAp7v1ZH1uhLL2wsd15+52V0Q3MKI1lgpqw/J55Qtqpj4IlHaDNYDTDSjRHuT+fz95hU7jePkszQ0eEkfn1oUTbOtRydbqV+Z/B7IoBqjgzXLpD3rD/q+2YJxEb40CWXttqNo6meYCXbxL524HiiVnZ8PsmKaaLrGV6XMROfG++bwwdPPoRaonImmjYyiGb/y5r2zE53b5uDNlvnNlFYGrvFKf1u0XyxQLb1GH312DCjgEUIK0cOdHOPsAa8sP5UWZNq0NirFJxBV9ZmYDoqaMtShn02KU8BTRbsBuJ9DlUmMnDy+/9r93uO+9Ibz8R83np/bvmPVKPZtWuI4I9+nXG3sIwVeOvkJCgS8VIMnfcysa5xMQpj7orrG5wEfTSKejjtWvDV60mOvbg813GUb2jif+ZYlxMlFgCKBXJc91AixYnTQ2P49kXZAJrVO76nU4Ik6XBoIqcH70e9cgRWjg6lgKDU2evKnfU+ravBue9OFuO2NF+DPX7gXQByJMQwUHn4ijm6ofdH8Jpr5IBISe765YNDvWaNUPhpulGrnzIUv+b7rJrQsoauqsJLmc2y2DG2gLYwaGjzDRDNuX5GGvsxadNwRZKWWaO/ktcprGhusWwE7dF0Rmkaic3e77Oa2PInOi5B58JpC+xWXn7xberGog4VLaaL5/i/ebeyTWldXlEsgfuf05snERDPdl45f2TuSCniOPHg2Mg9e5NDgtVpx4KisjPinT+A0gqw4Fq108cONGr6YfJ9yUTSthQIpIPcbs2cWRAiZlZy8IQ4S8NTda9JttubJ/hDZpINwlyU836TJF3FwJiia5G5eNoJ3PevErpYtN1WZVGqqaD31BPoFp23AeTtWeI+Tq75VSbUpaRv9x/QDerLq0spo9IRNr1Jr00FJsxWbIRWZiTVqgTPno32Krs8W8AY9GryyPuE00RR/K2tCZp8L5FfMtXmjxsyDZZYXKNMcUwqXWku9YCB/bfUwMEzGIsST3N3rF+HLb4y1iXrSrROD+xaNxgbrWDE2iEuOX4VvvuVi3PSU4xAEKg1rvyhpnzdNgiHg5Y+x39Wi6ICupNwyyIq8hrqhwYsPks9iotWqrMHTC1XjzZZRXzwOuK9F1pUG+Sh4v8uCl9gavIlmC2EYC2dSmyOPGx2sOYMCtaQGT/mDrNjvTStyJ58vYqLVyjR4ybn6b9vHvBMf6ED5zYMnpImmR8CTizg6yIpGF6tNzBUgTDRVLk2CjTSFdfngHZ1s4gcPPGG0RV8TEPcnWaphounyARUVpEKrnV7Dk5qlH6GARwgpZMvyBbj7nVfgvGOzCX1+Umeu4tucnQRwOG7NWFfb5qtvNmnwyvxKnrZnTeH+Ilwl23nwXLjmbZU0eDq8fA8+evo+pYlvXT54ffSxTScjhgbPPMbOTyUnU5rJCiaa9TDA0pF8WHr7Oeky7LKq+OC5CJTL7DQruyy6LpD3eSnqW3ZuwUApfOylp+HpyTskJ8Ablw0ndTu0EsIcTSkAiS+QMVlMBbzqPnhjg/VEUMm0sgtTDZ77HPkuVuneRcdEiHLPIw0WokwtsOmDl/yUAl6zWIMnF/z0RH58siXqS9orzpHCrKnBK8+DVyY4feo/77WOTzR41rtzWAQaGR2sWwtiWXt0+4LA74M3aAQRMcP+V0UHHgOyZ5Vp8OJjdBs78S0PlMpFvdYcOtpM2+wz0QyE2Xi82CQWrKzxqxVBmGgGxrjien+araxP2KkzAODZf/Ylp0WCNK32Blmp5y9alq/bnvPBy/lB5orpG2bPLIiQWUDV5JjzHV8UTd8E/Km71+Cbb7kYu9Yu7E07PJqK2UDZpG0qQotbM1Jetis6XpV7lk6se/DVS83lCiZI/fSxreKDl0uTEKhcf0lzhAUKn3nt2c66GpY5lMbuH3qCaK9uy6iTkrKFEuUQSF1BVlyPTd+XU7csNbYX9S3bRFMB2LB0GC86sClX95+9YC+uP3Mzdq7KLyrVwixiYxQB7/7sXbjz/oPmuxPEvlt2CPgqyGN1Prgq76LrCPttKGqGS+stTSYb4h2Xz/bVF2wHYKbOOGpFXLTZLHImSgEv88FzaWyy3+0omnEwFH99djqQMg6PN9MAK76cpaODZhRN3SlaUYRmFKV91Dc2ynQctTCIBZw20yQAWSAhfU/0c1JWf+/IRLNAMzrebOGRQxOpD6RLkJWabm1NIMsGMh88GRSlig+e9Nl0RbLVnLgunjfounXX1aalmoESDZ5En1eWB69fzTMBCniEGPznf7sY33nrJTPdjFmPPealCY0LxsKxQX8Otm61Q1Mlafd0UabtmorQ4jq3ivCVmVFl26o48OtJYS80eLaJpvzaL1tQPEmejbgiy+ZMNHWaBBFm3H58E81sUrVthTs1h55gv+3K4/Gh605Jt+d88GpuHzxf2pJqGjxzm5yEpusvHhPNz7z2bPzx1SeZdVbQ4Ony9M9NSUqPaw5sTI9dt3gYNz3lOOeinfbNkjx08KjTZ3FCBI2oiiyn3AfPfV52gL0hf8wzT17rPfzYRMDds2GR10TzeaduwN3vvMIwcT1cIuDJcUZP5McnMxNNn8+VRmpOdCChomdfRcALVHwtAPC57z2URtL0ab5GPT54iBLNebLBd74OoAPEfaWTNAmAQ4OX/J1+V1MTzc6+a0Xv8b2PHk6imALDDnNmqdmL0yRk6O1HUg2eyB9omWi6+nbTNtF03Lrnn7oBW5I0L3ZezaIomi4NnhGxNtACnhVZlBo8QuYmQ40wF1ab5LEnbNngPb2joW/SZAcamEnKhJIpafAc91tu8RWt10pTgS0oNo/S6EnuVMKZ+8j8KnQbM/7ulWfgXc8+sS/TJBRq8LQPXlNq8Mxjmkmi8yIWj8RCxItO34RTNmdBbOzbpSf19gKI776WCTUygING/l1LJ6h5Vo4NYNuK0VygIbu8Z56UCS+2iZa+LUtGGrj7nVfgGSety9XjDBzhM0fzCMS+cnyk0TzDzEzNd35ZkJUqGjw9kY7iAtPtO1aO4szty/CFXz8PT9u9xmuiqZGTXTtnmo0UxnSQlbf//XeNvHu5a5FBVqS5ZhT3QdkPX3fRMWbbKghOrQg4QViJPH54AmGgvAt+CwZCp8VDK4o1SqHjHZbI/lELVakGz7eQktfgmSaaWq7r1DJF9qsrLZeAex89HAvYgXIuwspFnGYUGc/Izq8XRxHV/sRmvlungNfKgqy4TDSB2EXETo8iTWnlozFMNENHkJUKPnj2QkI/LSrazJ5ZECGkbwmtAbgqZ21fhpueclzH9frG3lmVJqFskjwFocXlW2HmK3SXrecg+rlVvV9aIOmFoKUnU1mahOzDu37JMK7at77rdfYSfYuKIlGmaRKkiaZVzmTiH1PEkpEswIqhkfBoDKtOFMs1eHmBVE78fZr9fRsX439es89dpqjzq2+6EG9+6vGiPvtnZ/2wHuY1pXFD7ePy/oRV0Ne9cLgutI3uY9vV97i0oTLfmJ4wX7VvHW55RZySZP2S2JxS5lJ09QE52X3yaHUNnhbSb73zwbQvu56NL01CbIZsnvPK87YZ59pRMn3I5zTZis0sfRo4u/9KwaHZgjDRdJ8vr7AWBIhK0iR86Q0X4N9/47zcdi2YaPNAO4qmrsnnKyvZvmJBbpu8J/a9uPfRw6kvmz/XpU+DF/80TTQTLaS1WOVSPkpfR1eQFQCpH6Vui7webVqqkVo7l+beEPCSxtsCnX4HpjrGzAYo4BFCpkxZFE0fH77+1MK8TpXrtbb3OtH5bMF1u6sFWcnMaIDq+ZVSDV4PPnoyoiFQHhZ9tnHS+jja7IBlXmWmDTDPsX3wZG42zdGJVqkAvnhYRpL0C/h2dL4yygTBIMgLgfrvmjHBM495xslrjdQIxvlyRb4WGBND2+y1024YBkGp/6quX9fTzqKGbp/0aetWFE0F4I+vPgl7RcLzdGIrDl23eBijlkZGjosurZac7D55dDK3X2JMrMVk+t7HjsTtdFxuWZCV0CFsZW2rNiDY9QZBHEnzz1+4NycgyWdy+pal6f3/i8//EPc8csgZCddHI1SGiaJGBj9aMtLAusXD9qleE83U5yz56Uurs0xE0NX+qBJ5nYafWi3AvY8diU00HYs1cd2ZX7AdZEX3gSeSvmIEWbGtBFwCv/DBiyI4VzsCYU6dvY8qrU8W68vzqJG5WXVfswXyD3/5x/H+Cm4ns535MQsihPQU/f1wmQxOR702nYST7kfcZmbZ777nob9pqQavYnAhLaz0QoP34gObcOHOFbjGMUHpB15w2kbc+rpzsD/J86cnIXKeYwvG+u9xEZjAfqbjzRbGhor9V2Uyb9PkzDyuHrY3aamkwUt+12WmkVbFuXZ9Re+nFIgHaoE7aMtUNXiB29/LJxC3O55IE01f2RpX6HbJ6VutIDRK4am71+ClZ2ULY0OpiWaUml+7apOTXpdWSk52ZaoKF/L0gyJX4U+TvGW2vxSgk1nn65psRomJZrbfFjbsaKv+dpnn1cMAjTDAJcevwtOFuS+QmXN+4dfPw80v3p/esw/8x934l+8+gOOTqM++hQ7ZxFoYoNkyBbx6qPC1my4qbbOd6LxhvUO6nmGPgLd8NBPwXP1a3kup2VoxOoCHDh5NTTRdSLNxO02C1OzpnxOpiaZZntsHL4s22/Ro8EKVme7qtuuI3ns3LjY05D6N+2deezbeffVJePvTd+Xa4zP9dflQ9xt0NiKETJk4mt70OyT7xt6p+Ih97vXn9kRD1QtczXQGDchhBo6oaqKpTbx6cX8WjzTwvmv24/EjE/jhQ0/ghnO3dr2OXqKUwtblC9JnYpsV6WMkscYuyyMVqnwUTaA8QJEv35RrsgtU95QtC+oQONqrBbQiLXrRpKlh+YlNGGHSzfM77YU60bmN3axGzlSuGvp4OcmtkpPStSDz1qftwkvO2oILfv/fjDZK7ZyMVqjlC9eEvVGi4ZD3Wuf/c6GUeT2X7lqFy76zCv/07fvTxNS678nrk79LQWii2TLysLmeTdUomvZ9/pULt6fbZBTRO99+afreaBNW+9z//pzdAPwWDvJ51RMfvHbTJABZbsN8mgTdz+OfPhPN1QsHccd9jwMwBW8XDUuDJwM8uVDiu25HmrXHFxlkRt+zkUaIJ5OIpjYTzRb+66En0nNdPnjSRFP333OOWY7vv+MyNGpB+gRk4KSGiJILxMGp7ABV+pH6fCbjsa/FICuEEOIKENFrfP4tU2nH5mUj2LA0b0YzG3EGWVHu3yW2Bq9qdLYwXVmu3sZ2GRus489fuM8wO+on7GS8ciLtmuTUggBHm5mJpuuZSVM/F4OOiHGAP01CVUpNNB27XYF47MOKytXXokPcy3JsAaDT17wW+s3RJAMda/CSegrMczVlefAatQBbly/IHSP7RBpkJSo2ba6VmGhKbcYTHhPNMFD40e9cYdyrwXqIP3zuHgBSg5c/1+eDN95sJfnW/FoTOz+Zi+WjAznB9sDWZTgtScVRlgjbHgbXLBpKtpf7bNbDIJcHr6pFS6bBywKUAPk0CUOedCbrFw9l11DyUgyI664HgUgD4j7eDkgkH4N9TqsVpX6E+h3PhOd82Xc9+ASOTLSwfskQosidn9Vloil/l31GprUomwOUafB0+/tZg0cBjxDSFWZiIOznwbcblGnwfB+51AcvcIfN96E/oP0UzXK6SbVMDhMf1yQqCMwgK64+XWaiOVBzr+z7TDR1x/nI9afiI9ef6i23TLBRSuUECt1+qRXO+UUVvLdaS6En4KGjP9vpEtqlFgTOoA92eQ3Ln7IqLhPNKj54ZccCmdAgzXJlkBV/NjHzmbierdRmHCoJsmK3caAWYtmCAfzs0cNxO10+V6Jp48LkcqIZR4otMosrM9EcG6zh1tedU6hxSdOEhIFzDCvql1roatQCvDaJ8KlEmfUkD15TPs+K3Ub3E20CmY6z6euqNXjucXqtEPDK+qqhIa+p1ETSN6a7tPSaXMTfJH+gbMeGRMDTPpS3vfEC3PbGCwAAt3ztHgDAiWsXpefbSLN113dKN1suBlVZyNLjik8zXJviGDMboIkmIaQrhJYPxXTQv0Nvd3AKeNKPxXNeXoNX7U5q87t+MWGdCWwthNTiuCYLtSDA0cksmbZrklmmwVssAjlIykw0z9y+rLDcKnnwbNwaPGUd4x8otIBnC8qyzFRzX9g6P5efsArHrMxHG7TLS33w2tR8ukw0fRNFVx48tx9VfKzLRFO3M0r/cwsr6fNX7mc7UUGDl7bVcf7oYA0/f+Kot34pzEoBb3yyFU/QPRYZcduKBbyVY4MYG6wXjk36+v0+ddn2G84xTcRrgcI4gPe9aF96jUoB5x6zHP/y3QcSE80IzSnkwZtMgpjUPIKuz0Rz5ZgIHlLyzsrFIDn2+IRbGUXTxr7X0kRVn6MFvHseiQX/FWODxrNcOFTHtiTyZ8uhwos1ePHvLq2rEu9MOy4Huum+qKe2JUY/Qg0eIaQrxAEXpnc0tH0U5huuD6+8F/5JpbnKWtV0r+4IoEFMbLOqoiAr8X6VTng68cE7cd1CPHe/O4WEz0SzqnxexQfPRpvxmoKtdUxBsTpSoGs1P72nBYJAFS7dtRr/3/nbveVrOvXBSxdOKmjwis6X2JN+KfRrYSsSgSpcTdbPv+6JImpo8Dw+eOnY4TjfTGyt25EdJx/peDPTELaSMP221vuWG07HJcevBJAX8FaOmSbctvDvIo3C6Nmvt5+4biFuvOxYY19q/meZDb/76pPwD68+E6OD9bwGryJpFM1mrMFLffCSerRg7DPRlObsZRon08c10+D5BOPYt94j4Nk+eCIqpt53/s44IMrW5SOizOy8dz7zhOz6fRo8h4mmRhcVhtlxVb5nOm6AT8DzCdn9BAU8QkhXCNT0hxTu47G3K7guX94T35wyisz9VU0050JksV4jTYbin8WRFGuBskw082UWafBeff72gjxf5t/tBlkpk0l81wNY2iurvLBAcNRCgiv2gb2q3o1++MPfvjwNZd+1KJraTFWcV6UIZfUdo8xUgIn/luHuQ0OI8ica1ws0Pg2H6YNXkgfPUYQ0IWxHg6ePD63nu2/TkjR5vZ0m4bJdq42/s6i1BQKe5dtmo9vs0pRJoUsuLA7WQxy/ZiECBYcPXjVkonMFJfy/4v36Xvk0eNIHt8y6wkglEARpgKdMODLP941JgBVFNFCxiabOsZrsPLB1GW570wW4+PhV6bGyvMF6mN5Pl2ws+4VTwINLg1ftexYolfoM2oTWM+hHKOARQrqCz7ysl8x3X7CyQBE+zaZe59cfQpfpiwuX+R0xscPDy1vrkmtCS8BzPdMiH7yiZ2FP9hq19p6bbsvogFvAdL3uLrPfTKuZvyc2WsBzreb3YoEhCFRap12sfj/a7e9p/rQKUTQlhQKeJfyY0VKz43R+waUj+SBFmYmi+wEct3os/V1r8OwJv+0HKRlu5LWKkqYRWMV8vmGQvR+m32r8u9Tgved5J+HyEywBT9crqj3nmOXGMWUaPH2uKx1BLcz6tStHWqDyefCqdlPpgweV7+c6jYrdrqv2rcOutWM4bvXCdFs7UTTrNWEenrT14uNWGcfbyeAlsp8uGq6jJdJEBI4+qZHlDdSDQiFK3m93IvbkZ6DShaOy4FDpuYFyBnaJ6y1eDOgHKOARQrpCGEy/oaRd34U7V3hzBc1FXB/GSonO9SprgemLi17mwZsrZJPU+GcVDd7RsiArBSaaRcKHXVYnk5Y/vvok/MOrz3KX7/Cz04KIy7xTH1+kwdNaCle4+W7lwbPRicK7FmQl9cErfvY2+h66NIbZvfYv6kQArjtzM/7ouXvwDCvnGyBMND0T4N9+5gn4/SQ1gE50Pmhpjdo10ZSh7ws1eFIzZmmGAFPAW71wKDf2pX0i2TFYD/DB604xjtECwkGPf6E213MlFK+Jsc9pGq8UWq2pmWhOtuKw/DrFgL5GrVm123XalqX4+1edZWwv62fy2detsQcAfv+q3fjc68/NyivQ4Mm6Fg7FJqq2iWYZA7WwsM1yTHRp5jJf50yDV3XBsqiJmftC/37rGGSFENIViqJt9bJOIPu4v++a/dPbgBnGrcGT+93n2R/hynnwtA9eH69q9ho9SdfzPHmvnJqZMNPgycThkrECE80i80GfRqqdp/fU3Wu8+2TVWiscOvw0M4Es/lnUf4Ya/smZ7aPVrW6oo1Dat7LTNAmBY6JZZcJrCymSohxxWmMRRfFxV+7JC3fyOJ8J22A9xP5NSwAATyYmmkP10EhkbrdHUiZoSBNQp4mmvsYg/87I48NAwbass4MbufrYOcfE/mA7haZScnhCX3P+fZMavEyLKutHLshK1SVPHfgkVuCpXIh+LdzawrYMmLJ20RB+9ujhCoGRsv21UKVpEvS3ZLAeYvMy6S9XHIBFs2i4gXsfPZIuzFT9RgzWg8L3WKZJcC0aycWezPe1Wt1FbcxMzftXD0YBjxDSFWYiDx4VSXnkM/AGWUl+tq/B05P3zts319G33LWS7ZycB0GqOfBp8EY8JpJAsTbVnuy5JqZTocgHz5UmQU+oiiah9iTWrC/56TDlmwpag5fzwevQRDPLg1f87G30IS6Bsihse9X7UBZFEohD5wOZiabvebiqlD54Ku1rKl3tMHLfTZo+fqHhg2cKIQBSTRMQ3ws7Kba9eOB6ZqsWDuIbN13kjch5eDxuk9NEUywupCaakM83ThkyFQ0eEL/rdqLzTMAzB175jn329ecgioAv/dcvCuuSfaseBjkNnk2oCqJoinMWD9fRirIgK1WtPAbrodP0Ura3SHCTvpdt++CJNl5w7ArceueD6d/tRpiejfAzTQjpCj7tQ0/p37G3ZxgaPM8xenLU7gdxLkQW6zV6YqDneWV+WPJ51QK3FtwXXEGf48OuL42w2KUXRynksq6lqTQc5pvBFAU8WwDoVjfUE0y7vG6aaFZZ/LJ9FY0yU6HCUV/F9tVLNHhApiXRaRJ8fc+pwasXa/Ammq30Ho83HSaaDi2lFrKlgOcK3W9Hr/VpXhaPNLBibNC571CBgCf9MZ059AJ07oMnBJwty0aEJjPepv0VbdND+RwHamEcsKSkL4Q+Ac/TWFVgmSOrWjLSMK6/ugYvxI5Vbo1qXEdWjqvPSb/VdqJoyrKVAv7yxab1T7vfxtkINXiEkK4QBNM/8c+imRGNfAa+53Hq5qX46t2PYNXCIQCPtO2DxyArfvSd0SvZRrRDp2ZG+Gk5NBOA2yeoqMy0LT1+TEUaPKcwm2qA/Q0rEmbtBOfdshjIgqxYGrwOBTxdTrv+O6kGz3GeS7v1udefiyeOTOLgkYlK5cs0CT60luixw4kGz9P3nD544ljX/slmhHoYYHyy5TDRlOkvpAYvbo88vhYq1EKzXbZ5aydj1OECraURZMWxwKCUjiLZgQZPCBGbl43kTTSTa7fH6aJ+4kOeUwuUCIpinjdUD3F4oplY5rjLkucsHmmgKYKsVL3/g7UAu9aOJSau7vZq016nb6oQ8NrJgyfb6Etfo+vvV/pXNCWEzCp8Obx6CTVJecxJh/uYX73oGPzr68/F5qVxEtoiExlJzeFfRUx0n9STFekn4+qvDSvcvUtoKTZhyu/TPnv2c4oyFV5XkFr7NMhKwSq7nX/RhUvA09djp0moehm71y/yRgIFsvtrPx+dTPzopNukz0eRz1CubeuyCIgQk1WbNNed0JluXjaCE9YtrGwO50phYbNgoIZ6qNJk3kN19zU4o2gK3zVXX2+2olSYyUXRVJkGRk64dZulxi8MVE7LZvvgdWJapwXgpQsauX0yyEqWa9BcTIsidJQmQUa33bx8JL2PuixtoinHEtleie5y9VDhjrddmt/vEJ6BvNm9g/0R7QAAIABJREFUvr/NVlTJB2+gFhpBVqr2yYF6iOFGDVuXL/DW0UyigrneJ71Nzj+qa/Dy12GXSxNNQsi8ZyYiK/bv0Ns7qkTRDAOFTctG0glKlYlofJxfO0NiVCrg5TV4rndkoUiBEARwavCKNFWuR7dm0VC8L6eRiv9eVJB2oR1cr7wrhLz+tYpuw6WtPG3LUgDAI4fGk3rzdRTxv3/5DHzrrZd496caPGv74uH4PlXVkGnS4EUV0lJ85CWn4oZzthr1F2kUnFqOimOvUgr1UBVOgJVSWJLkBVQKaNTcGjzXGCAD5ChHFZOtVipcuqJo6mbJy3G1NVTKSMkgz9GHd7II9dKzt+DGy47F1adsyO2Tic5dhCpvolkVOf5uWjqCZaPx/X/4ybi/awHP1uC5NFXSB7EoGigANArMx19xbtwnRwb8US5DQ8BF0tY2TTSTa9q2wi3g1QKV+m66FiZS/9lAavraN9G0mQs5XyngEUK6QpEzdq+wc47NVv7t187FV9904bTUZU6si+9L21HHpmD+NF/Qt0YLaq7Q8RIp4NWCoO0Jomsys2qh28fo5A2L8abLd+J3n3ViW3X4cL13aTAQ0ff0cVXiT7jM4158YBMAYMOSWOPcrgavjEyDZ25fOBRPtF1RJItIw7pXmGiODtbTyW2Rj6Le5kwf0cb7WA+DUhO2JUkOvUYYwHeoS0Ys88GbbEWpwGYLeKFh+ii1TO57YWvwdH+rkvDcx2A9xA3nbHUKldJsUr/bsoY0D16FTn6FlcNPtnXl2ACWLYjvvxbwtLbT1uQXpSLxIesqSuPxkrO24O53XoHhRs2fJkFUr5+dThxeUcZK27AhsSZxtbfI7HNILM5o4bJqvk9b62u2q/+/dfTBI4R0hSJb/V7hWiWejWxcOlJ+UJeoosHLjo1/VtUA1AL3RJhk5E00TRNMm4XDdWP/ZJsCnmsCsjoR8H6RmNlplFJ46dlb2iq/iEC5gqzkpa9Ug1dh8mtHCgSAA9uW4btvuyTV2nR7YccuV6M1eDr4RlWyQB/ttU/fHdd5tQIBr520JbWgWIMHAEsTDV6jFhjpGVpR1kbXpLhsMWOyGaGemlya9zRQmXAihQOXkFwLVU7YSYNtTEHAK0IGWclMNGX9SR48I8iKuw1/8vyTcfZXf4Lf+MS3jLKBOEG9LqFMg+fqJ0HJYkotVPj4Dadj1dggPvKVH6fbi3153fsMf+/k/HY1eJpNnm9kIMZE13dKC3iTrVZ6n6pq8IoWLOfCYmafTI8IIbOdomhbPatzeqvrC6oEWbGpmuvHdv4nefTcQk+wBozQ8Q4BzzLR1BPXY1a6TZZs3AJebKJ572NHKpXRKUVBVgza6C6+JMXSJM/2xZsqy0djjclRK7Lj4pG8L1YV9BzfJUhtWZ6fyOrLaKWaCn9C58lW3h+wnUlooxaUvu/aRHOgFngDzXSUB6/VQr3m1uDJyJiuNAlG3Q5f1W744BVRZqIZqHgBo6oPnnzGRrqBkTqWJxo8LdhorZjdnxYN5/tnGsXXU2+gFPZvWoL1S4aNd61oTPfdSleOT+0r2a5gtHGJW4NXC4s1eHpRYXyyldZd1QdPN9916VUi/s52KOARQrpCOANRNNv3dpj7yO9R2dPQ89mqk6FVY4OohwprFw911rh5gJ54NlMfPBFZsMQHr1ELsG7xMG5+8T784S+d5Cw/b6aVL/M5+9ZhzcJBPGfvuvYvoA3c4fzz04pXnrsNALDSYzoq0fdvk8dkC8iE6G6le9ATalvjuWi4M19FXzCZf339ufi7V57hPa9IU6H9EBc6/CfbmYTWgiDVovnQAl4jDEQSeLdAJRkuEfAeOng0C5riMtF0+D35TDRt7Mm6qx9OBS0UB0JtnTfRtDSsBbfZzEeX/b5goJaaaGr0vZLH3XLD6Vi7KD8Ol0bRlCaaHiHTxhtkRZyjf9V+cO365J+0YTF2rBx11j2ZBlnxa/AmmlE6NrqC5Lgo8rNrFryL/QJNNAkhXaFdk4xu0r9DcPcxE50XHztZEJ3MxZpFQ7jjbZdW1vjNR2wfPDlxcppoigm7FgbPP3YlfvrwIWf5X37DBTg00cQZ7/xsrnzN6oVD+I83XNDZBbSBOzhBsk9se/bedXj23nV42nv+vVK5//Sas7DKk6ssrlebaFZtaTFag/eLJ8aN7YuGOtXgJZNcq32blhWbahflEHvTFTvx/FM3YN3ivODbjoBXr1U30VRKpRpou45OTDQfOTSBkSSaaS4PntDgyaJdJprOKKN6YaVHE3OXBs+IounIg1eEMS5YZdrBUVx58PZtWuIst2woN33wZBv855Qt3J61fVkuKXu7i71DjRD//KtnY9ON/2Bsl6kcXEK7Ts0x3mzhKSeuwQOPH8ELT9tUqc4iS4DsHe7f2QUFPEJIVwgC5Yzw1kv00LtgkEOZxtDglXyciqKT+aBwV0yRD4zLZ1RHtByz+rBvBXzxSAOLxd8zucIcKJWuoC9IUzO4k4ZLylq8c7U/8bGut6yOdtAC3s8tDV7V/JA2WnapOjnUh+kFF5cAUw8DbHdoONqpBwCWLRhIozT6WJkI1z979HC66GALna42jjjMaG3uS8yGxyfNF8SMotm5Bq9ZIRVHJ0gNXuSwHVEuDV5ReYYGz93PFiTCsO4XVUwP03tX4IOnkQJj0beiqHt98cbzsXSkgY/d9hMAwITuw116OcOgOA/ekDDRDAOFl529Nd33v375DIwWzA2yyKt+DV4/m2hyVkQI6QrxgD69Et7oYB03PeU4XHDsimmtdzbTlgav6Td9IZ3x8nO24rv3Po6n7V6T21ekwbP9aao+kplIT5LWrYCn7l6D+x47gmsObATgjqKp+f3n7MYf3foDHLemWICrUm/8szvXviIR8B451F46BB+drv53Oqls5/ibr9lfKrhesDMbT7VWzu5nLllDRm/1Xbq+RluDF6osiqasyhnR0qHJSU2je6TB02attSAOpmK3M/XBi+Jcf+PNVokPnluDp/n8r52H4YH43r/1acfj7X9/R6GwYpflEkIBK8eg1OAV+eAV3Mu1VkqWzESztKmVKIuiqYMy2Sa/ALBn/aLSsgG3cKsXP2dyfJ0qFPAIIV0hCBRazWlW4QG4/szN017nbMacdFTT4PXzKuVsY+2iIdzyigPOfUWJzm1/r8oBcmbw2cWRc1WaM0tv87F95Sje87yTu1Z/tzR4SwqCqbzj6bvSqKRViTwmmmV0+j62oy2pEjhm6YIBXLVvHRYM1FO/JrsK16R4ufAdK7MeGJ8siKIpzi1K5m2fD2QCRvc1ePm2SV10oBR+8eQ4vn//QdRDhbLAq1K4cr3DMm3AlXvW4so9ayu1syyKpi9NQrEPXvb7aVt8pqGmiWY3NXgTBZYmmQ9eXsAro9BEkz54hBASEyqFZh/bq88VDA1eybFzwZG8n3BNTLcsj6Nlvur87cb2qq/STArnhXnwetgsbQXXLQ2enuheuHNlbt8LTtvYdnlp+yo+m9S8sMAUrYiwzXQMVXjXs3cDAN77r/8FIGtbWqfj3rej7fjyDx/OnevKg6fT70iBxanBE8cD7mA0U6Em0iScuX0ZLty5Ar95xc6s/Umb733sSFJ3s1DIlf5kSilcevwq7N/sFp7aoSyKprx3MthOcRTNeN9ALcBfv+x0d72WD163xqVAKbz87C344l0/x5nbluX2D6ZpEtpfXC7Kg7diLF6smMnYAlOFAh4hpCtcuWdNR6topLuYPnjFx+qcRfSrmx5cE4klIw3c/c4rcturRoicbdrXYBoEvCr59Nrlzrdf2rWFDm2i2W6evlO3LMW9jx7G6y/Z0dZ5vZyEpiZwtkllh/dq5+ox3HHf47ntoVJCg2fuqweBUb+rav1u7d+0GL9x6bF47v71HbXPh+4boVKo1wO875r9xn4jMExY7oe6csyMlPlnL9zblXaW9QW5GFCvqMHTRRb5AOrTxycjKFXe9z/xigO5ZPUuaoHCrrUL8fWbLnLul4F92iVw+HwCwN++/HTcef/j+Mdv3U8TTULI/OPD159iDK7PPWXDDLZmfvH+a/fjGz9+xLmvnTx4zYLw06T7tDMRz/zMSsqcZc+uyAevW0Rd1uABU5so2viiaEpc2oiRgRD/+Jqz2q6vyxkBDHSQFVtD0um9//tXnYmtb/zH3HZfmgQgNs2TJo8u4SGLrGqaDHeLWhBAKb+WUra5UUGj6kvsPVXK+oKRJqFiFE39LheacSb7JlutSuPc3o2LS4+R5fqwI462Q+rzaV37KZuX4HsPHDSO6Uco4BFCOuKs7ctnuglzhqftbk/7ed6OFThvhzuwTDsTr0lq8KaVduYK+jmWBcSomuJiuphOgXO2zr10EA7f5NDWFk5VGJ4ODZ42506Fa0+3+80rduLjt9/jLc/XP5TyL2r4FqBWjA7gwYNH0/N7Sb2mChfCfL5tPrq5oCDJfPDcWm6ff2ORxk0HbKkXCK7SRLObWq+yxcehKdxHfc2ub6beQg0eIYSQjnn31e6k1p0gv1WlUTTpgzettGuyB5iJ0l3Mtkc3HXmj9IRzll16SlkUTXtyf9b2ZVi2oIGXnb2lo/p6KVT7BBGfUPmSs7bgJWe1fx1hIPPgmWX7TAP/z6vOxHs+exc+/OUf9/w9ePqetVg5WpSbMftdC0Iz0T/LffA8JpoF722WpqDARDPZNT4ZdXXBoaxv6wWIqZTtek/1O9zP30YKeIQQModox0QzTXTegyANZGpoje5AiQavE6FxOuhpkJWWrmN2Xru2ZqzavKULBnD7b7p9jKrQSwHP7n/6mrqt2QiVNNE09/nGp5Vjgzh961J8+Ms/7qlJMBD7DhblZ3Rpxsr65289Yxe++dPH2mrH2kVDaX68sna4kD54pomm/zzdB4rSDqRpElqtrvbHUgGvZAGsiCIz+F5FY51OKOARQsgcQn7fSwW8CiuzZGZYtmAAT929BtedsWmmm9IW05EoRdcxW+deUYd58DplOjV42vKvEy1NUTOVKvDBKxifUm3pDA9hssVlZtWa55+6Ec8/tb16vnjj+YX7y/qCocETN62ory5dMIC/e+UB7Fg1WlrvRLPV1feyrJ9NZaGhKIomE50TQgiZVciPVdnHiWkSZi9BoPDHXTTdnS6iDiNItkMr9S+anf22Oc1CRy8FSZ8JXCd1Fglq0kTTLlv6fm1bscDYl2lLZ7YvNEUQmhnNTVmSB0/e26oaPAA4aUNxUJTUB28y6qpQ1EsfOLu//dNrzsJjhycAZO8wBTxCCCGzAv3RrhIm/PydK3DrnQ/mJk2EdMq0aPDSKJrTUFkHdDtPXxm9NdF0m8B1IrwWtTMMsv32bdMmj888eS3+4Ko9xr50QaH95nSVo5NZkCwdZGUm2qTaiKJppkmYYr1awOuyiWYVnnXyOhzYurTt8+z+Jk1wtcA+Xe9wL6BdDiGEzCG2Ll+Av3nZafitZ5xQeuzzTtmAb73lYmxa1puQ3STmVedv63ri5Znk5edsKfQDAno7uc20hD2sZArsXrcQALCxR6HwbaYjimauzg4m8UWaLaWyROf5NAmBc7tkpifiRyezPA41j6A6HZTmwTMEvOz3qWpATRPN6b3w379qN561d13b52ntoKsvt+aAdQs1eIQQMsc4dUu11UylFEYH547gMVt53cU78LqL20tePZt5w2U78YbLdrp3ToMKL/PBm52Tr+vP3IxzdyzHthV+n6Vu0kszNp8GrxOhMrSCpYSBynydZKJzS6bUgoirzio5B6eDoxOZBs9nHjkdlAneUpCTz3aqiwRaA9htE81eUi+IopmZWffHtbigBo8QQgghXSFNYdDDedGpm5dgwUANLz+n+wmtu4FSatqEO0mjB/ksfVFcO9H42NqQ77390vT3ONF5/Hs+yIpb8ANmT0RVaaKZ+eNNf5vaWfQYGRAC3hQFGR3N8omjk7N24cXGF7UVgLHw0K9Qg0cIIYSQrrBm0RAAYG9JUIapsHTBAL791kt6Vn4/8rvPOgF7Ny7perm+iJCdCAT2OTIheKD8efCKTDR3rY3NYS86bmXb7ekmhoA3gyq8dp7LSCMTAaaqqVo4HFuCPPzkOJaPDkyprOlC+6u7FgeyKJrT2qSuQgGPEEII6TMuP2EVPnPHgzPdjBzHrhrDZ157DrbQr3Na+aX9G3pSbruJzosoTJStZFRDc5820XQJeDtWjeJ777jUa0o6XUgfvKHknl25Z820t6MdOW14oHsmmouGGwCAwxPNvjHRDJP+6GrvXDDRpIBHCCGE9Bl/+vy9M90EL4zKOncYrIf4+A2n4xdPjOOGj3wt3V4WrTE9TvxeHEWzPA+e7/yZFu4AU4M3UAvwn2++uDQQUS9QSuH8Y1fgeaeUC/wNQ4M6tXoXiSBS/WLVWC8y0Wz2f5CVnikflVI3K6UeVEp927NfKaXerZS6Syn1TaXUyb1qCyGEEEIIaZ/9m5Zg0bAZjKkzDV5BFEwjD565Txs8jg7OXp3EuBDwJlsRFg7VZ0yTdfOL9+PCCiar0jRxqpqq4UZYGAxnNpKlSci3d3IOpEno5dvyAQDvAfAhz/7LAGxP/p0K4L3JT0IIIYQQMkuwJ7rd8MGzy/dNuO979DAAYMvy2Wv2KzV4rZkMo+nh337tXPz04cPe/VMVypRSWDjUwM+fONo3Jpq10K/B08+QGjwHURR9HsDDBYdcCeBDUcyXASxSSq3uVXsIIYQQQkj72PPcqpoNKeoUmmgqGUXT3PeThw8BADYvm72mv1K7mEXRnD1sXDqCM7cv8+7vhq+Z1vJORev1jZsuwtdvumjKbamC7o9OH7w0yAoFvE5YC+Cn4u97km2EEEIIIWSWYM/ZO5n31sL8SbqcIJBBVszjtHZs8ywO3POeq09KI3lOzkIBr4xuCDLaD28qZS0eaWDJSGPKbamC9u10CaT6We7b1P3ItNPF7DVoFiilXgbgZQCwYUNvIkURQgghhJA8ttlk1Um8GWQlnlB/4Nr96fmNWoAjEy3DRNOecG9bsQB3PfgEFg6ZfoCziRVjg7jujM349HcfQKsfBbwu+JqlGrw+0XrVCnzwzt2xAj/6nctnPL/iVJhJAe9nANaLv9cl23JEUfQXAP4CAPbt29d/bw4hhBBCSJ9iC11TSXR+7o4V6bZGGAt4YaBSIcMu+hM3HMDjRybarm+6adTihs9kHrxO6YYcM6Y1eH0iE4UFPnhAZ318NjGTJpqfAvCiJJrmaQAei6LovhlsDyGEEEIIsejGVNel9RtIcsYFSqWaH1uYXDhcx/olw11oQW/RGsq+1OB1QeumE6f3i9+aXnDol6if7dIzDZ5S6mMAzgWwTCl1D4A3A6gDQBRFfwbgHwFcDuAuAIcAXNurthBCCCGEkM7oNHBGPQww2YqTgLsiEg7UtB+UP9F5v6AFhb70weuCkDPUyIT1qXLZrlW4//EjUy6nCO2DN0flu94JeFEUXV2yPwLwy72qnxBCCCGETJ1OJ8GffOUB/M8v/BCf/PrPcOHOfG62RiLgtaJIRNHszxl3Ii/MyiiaZXTDb26w3j0B770v2DvlMsrQCw59aFFbiZk00SSEEEIIIbOcTiftO1eP4Q+u2oNv3HQRrj1jU25/I5HqdKAVoH99n7RGaDbmwfPxqvO3da2swXp8/ZOtVsmRswPtg9dPz6sd+iKKJiGEEEIImRlcMtfbrzwee9YvrnT+Yk/o+2NXjeLO+w9isB6kE+1+NdHMEmf3zwW87uIdeN3FO7pS1lCiwRuf7A8BT2vw+lDhWgkKeIQQQgghxItLaHnh6ZumXO7vPPNEPG3PGmxbMYpv/+wxb139wJZlI7jhnK147v715QfPQVIBr9kfElM/alzbgQIeIYQQQgjx0iut2lAjxPnHxr55aR68PnUeUkrhxsuOnelmzBiDqQavOcMtqYbWuM5R+Y4+eIQQQgghxM90KNXCgsTTZPaTCnjN/jDRDIO57YNHAY8QQgghhHiZDqGr39MkzHd0moT+88GjgEcIIYQQQuYZ0+EXF3oSnZP+oP+CrGgfvBluSI+ggEcIIYQQQrxMh1ZN10EBrz/RaRL6RsBLffDmpoRHAY8QQgghhHhRmD4TTcp3/clQ3/rgzXBDegQFPEIIIYQQ4mU6g6xQg9ef6CArE32TJoE+eIQQQgghZJ4STIONZibg9bwq0gO0gNcv0AePEEIIIYTMW6ZD5sqiaFLC60d0FM1+IaQPHiGEEEIIma9MZxRN5sHrTwZr/SVSaBPNOSrfUcAjhBBCCCF+psNsMmQevL6mFvaXSKEXFJpz1Eazv54GIYQQQgiZXqYjTUIyI6WJZn9z+palM92EStRD7YM3NwW82kw3gBBCCCGEzF6mN9F5z6siPeLrN12E4T7xxQvnuIkmBTxCCCGEEOJlOgS8LA8eJbx+ZclIY6abUJnUBw9zU8KjiSYhhBBCCPEyHVo1RtEk0wl98AghhBBCyLxFTYMTXj1UeM7edTiwrT98uEh/o33waKJJCCGEEELmHWoa1AFKKfzec3b3viJCkGnw5mqQFWrwCCGEEEKIF5pNkrmGTssxN8U7CniEEEIIIaQAindkrqEXLeiDRwghhBBC5h3U4JG5hs67OEctNCngEUIIIYQQP5TvyFxDp+OgDx4hhBBCCJl3UINH5ho69cccle8o4BFCCCGEED+U78hcY6AWAgDWLxma4Zb0BqZJIIQQQgghXqjBI3ONJSMN/PkL92L/piUz3ZSeQAGPEEIIIYR4CSjfkTnIJcevmukm9AyaaBJCCCGEEC+KGjxC+gpq8AghhBBCSCEvOG0DnnrimpluBiGkAhTwCCGEEEJIIe94+gkz3QRCSEVookkIIYQQQgghcwQKeIQQQgghhBAyR6CARwghhBBCCCFzBAp4hBBCCCGEEDJHoIBHCCGEEEIIIXMECniEEEIIIYQQMkeggEcIIYQQQgghcwQKeIQQQgghhBAyR6CARwghhBBCCCFzBAp4hBBCCCGEEDJHoIBHCCGEEEIIIXMECniEEEIIIYQQMkeggEcIIYQQQgghcwQKeIQQQgghhBAyR6CARwghhBBCCCFzBAp4hBBCCCGEEDJHoIBHCCGEEEIIIXMECniEEEIIIYQQMkdQURTNdBvaQin1EIAfz3Q7HCwD8POZbgSZlbBvEB/sG8QH+wbxwb5BfLBvzC82RlG03LWj7wS82YpS6vYoivbNdDvI7IN9g/hg3yA+2DeID/YN4oN9g2hookkIIYQQQgghcwQKeIQQQgghhBAyR6CA1z3+YqYbQGYt7BvEB/sG8cG+QXywbxAf7BsEAH3wCCGEEEIIIWTOQA0eIYQQQgghhMwRKOB1AaXUpUqp7yml7lJK3TjT7SHTi1JqvVLqc0qp7yqlvqOUek2yfYlS6tNKqR8kPxcn25VS6t1Jf/mmUurkmb0C0kuUUqFS6htKqb9P/t6slPpK8vz/RinVSLYPJH/flezfNJPtJr1HKbVIKXWLUupOpdQdSqnTOW4QAFBK/WryPfm2UupjSqlBjh3zE6XUzUqpB5VS3xbb2h4nlFLXJMf/QCl1zUxcC5k+KOBNEaVUCOBPAFwG4DgAVyuljpvZVpFpZhLA66IoOg7AaQB+OekDNwK4NYqi7QBuTf4G4r6yPfn3MgDvnf4mk2nkNQDuEH//LoD/EUXRNgCPALg+2X49gEeS7f8jOY7Mbf4IwP+NouhYALsR9xOOG/McpdRaAK8GsC+Kol0AQgDPBceO+coHAFxqbWtrnFBKLQHwZgCnAjgFwJu1UEjmJhTwps4pAO6KouiHURSNA/hrAFfOcJvINBJF0X1RFH09+f0g4knaWsT94IPJYR8E8PTk9ysBfCiK+TKARUqp1dPcbDINKKXWAbgCwPuSvxWA8wHckhxi9wvdX24BcEFyPJmDKKUWAjgbwF8CQBRF41EUPQqOGySmBmBIKVUDMAzgPnDsmJdEUfR5AA9bm9sdJy4B8Okoih6OougRAJ9GXmgkcwgKeFNnLYCfir/vSbaReUhiGnMSgK8AWBlF0X3JrvsBrEx+Z5+ZP/whgF8H0Er+Xgrg0SiKJpO/5bNP+0Wy/7HkeDI32QzgIQDvT0x436eUGgHHjXlPFEU/A/DfAfwEsWD3GICvgWMHyWh3nOD4Mc+ggEdIl1BKLQDwCQC/EkXR43JfFIerZcjaeYRS6ikAHoyi6Gsz3RYyK6kBOBnAe6MoOgnAk8jMrABw3JivJKZzVyJeBFgDYATUthAPHCeICwp4U+dnANaLv9cl28g8QilVRyzcfTSKok8mmx/QJlTJzweT7ewz84MzADxNKXU3YtPt8xH7XC1KzK4A89mn/SLZvxDAL6azwWRauQfAPVEUfSX5+xbEAh/HDXIhgB9FUfRQFEUTAD6JeDzh2EE07Y4THD/mGRTwps5XAWxPols1EDtCf2qG20SmkcTX4S8B3BFF0R+IXZ8CoCNVXQPgf4vtL0qiXZ0G4DFhakHmCFEUvSGKonVRFG1CPC58Noqi5wP4HIBnJ4fZ/UL3l2cnx3NVdo4SRdH9AH6qlNqRbLoAwHfBcYPEppmnKaWGk++L7hscO4im3XHinwFcrJRanGiIL062kTkKE513AaXU5Yh9bUIAN0dR9Fsz3CQyjSilzgTwBQDfQuZr9UbEfnh/C2ADgB8DuCqKooeTD/Z7EJvcHAJwbRRFt097w8m0oZQ6F8Droyh6ilJqC2KN3hIA3wDwgiiKjiqlBgF8GLEP58MAnhtF0Q9nqs2k9yil9iAOwNMA8EMA1yJeeOW4Mc9RSr0VwC8hjtL8DQAvQewzxbFjnqGU+hiAcwEsA/AA4miY/wttjhNKqesQz00A4LeiKHr/dF4HmV4o4BFCCCGEEELIHIEmmoQQQgghhBAyR6CARwghhBBCCCFzBAp4hBBCCCGEEDJHoIBHCCGEEEIIIXMECnhmHKf1AAACP0lEQVSEEEIIIYQQMkeggEcIIWTeopRqKqX+n1LqP5VSX1dKHSg5fpFS6pUVyv1XpdS+7rWUEEIIqQYFPEIIIfOZw1EU7YmiaDeANwD4nZLjFwEoFfAIIYSQmYICHiGEEBIzBuARAFBKLVBK3Zpo9b6llLoyOeadALYmWr/fS479jeSY/1RKvVOU9xyl1G1Kqe8rpc6a3kshhBAyX6nNdAMIIYSQGWRIKfX/AAwCWA3g/GT7EQDPiKLocaXUMgBfVkp9CsCNAHZFUbQHAJRSlwG4EsCpURQdUkotEWXXoig6RSl1OYA3A7hwmq6JEELIPIYCHiGEkPnMYSGsnQ7gQ0qpXQAUgN9WSp0NoAVgLYCVjvMvBPD+KIoOAUAURQ+LfZ9Mfn4NwKbeNJ8QQggxoYBHCCGEAIii6EuJtm45gMuTn3ujKJpQSt2NWMvXDkeTn03we0sIIWSaoA8eIYQQAkApdSyAEMAvACwE8GAi3J0HYGNy2EEAo+K0TwO4Vik1nJQhTTQJIYSQaYcrioQQQuYz2gcPiM0yr4miqKmU+iiA/6OU+haA2wHcCQBRFP1CKfVFpdS3AfxTFEW/ppTaA+B2pdQ4gH8E8MYZuA5CCCEEAKCiKJrpNhBCCCGEEEII6QI00SSEEEIIIYSQOQIFPEIIIYQQQgiZI1DAI4QQQgghhJA5AgU8QgghhBBCCJkjUMAjhBBCCCGEkDkCBTxCCCGEEEIImSNQwCOEEEIIIYSQOQIFPEIIIYQQQgiZI/z/jbMmtic9z3YAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" ] - }, - { - "cell_type": "code", - "metadata": { - "id": "OSatWxYzU6Zq", - "colab_type": "code", - "colab": {} - }, - "source": [ - "" - ], - "execution_count": 0, - "outputs": [] + }, + "metadata": { + "needs_background": "light", + "tags": [] + }, + "output_type": "display_data" } - ] -} \ No newline at end of file + ], + "source": [ + "# BERT fine-tuning parameters\n", + "param_optimizer = list(model.named_parameters())\n", + "no_decay = ['bias', 'gamma', 'beta']\n", + "optimizer_grouped_parameters = [\n", + " {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n", + " 'weight_decay_rate': 0.01},\n", + " {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n", + " 'weight_decay_rate': 0.0}\n", + "]\n", + "\n", + "optimizer = BertAdam(optimizer_grouped_parameters,\n", + " lr=2e-5,\n", + " warmup=.1)\n", + "\n", + "# Function to calculate the accuracy of our predictions vs labels\n", + "def flat_accuracy(preds, labels):\n", + " pred_flat = np.argmax(preds, axis=1).flatten()\n", + " labels_flat = labels.flatten()\n", + " return np.sum(pred_flat == labels_flat) / len(labels_flat)\n", + "\n", + "torch.cuda.empty_cache() \n", + "# Store our loss and accuracy for plotting\n", + "train_loss_set = []\n", + "# Number of training epochs \n", + "epochs = 4\n", + "\n", + "# BERT training loop\n", + "for _ in trange(epochs, desc=\"Epoch\"): \n", + " \n", + " ## TRAINING\n", + " \n", + " # Set our model to training mode\n", + " model.train() \n", + " # Tracking variables\n", + " tr_loss = 0\n", + " nb_tr_examples, nb_tr_steps = 0, 0\n", + " # Train the data for one epoch\n", + " for step, batch in enumerate(train_dataloader):\n", + " # Add batch to GPU\n", + " batch = tuple(t.to(device) for t in batch)\n", + " # Unpack the inputs from our dataloader\n", + " b_input_ids, b_input_mask, b_labels = batch\n", + " # Clear out the gradients (by default they accumulate)\n", + " optimizer.zero_grad()\n", + " # Forward pass\n", + " loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n", + " train_loss_set.append(loss.item()) \n", + " # Backward pass\n", + " loss.backward()\n", + " # Update parameters and take a step using the computed gradient\n", + " optimizer.step()\n", + " # Update tracking variables\n", + " tr_loss += loss.item()\n", + " nb_tr_examples += b_input_ids.size(0)\n", + " nb_tr_steps += 1\n", + " print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n", + " \n", + " ## VALIDATION\n", + "\n", + " # Put model in evaluation mode\n", + " model.eval()\n", + " # Tracking variables \n", + " eval_loss, eval_accuracy = 0, 0\n", + " nb_eval_steps, nb_eval_examples = 0, 0\n", + " # Evaluate data for one epoch\n", + " for batch in validation_dataloader:\n", + " # Add batch to GPU\n", + " batch = tuple(t.to(device) for t in batch)\n", + " # Unpack the inputs from our dataloader\n", + " b_input_ids, b_input_mask, b_labels = batch\n", + " # Telling the model not to compute or store gradients, saving memory and speeding up validation\n", + " with torch.no_grad():\n", + " # Forward pass, calculate logit predictions\n", + " logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask) \n", + " # Move logits and labels to CPU\n", + " logits = logits.detach().cpu().numpy()\n", + " label_ids = b_labels.to('cpu').numpy()\n", + " tmp_eval_accuracy = flat_accuracy(logits, label_ids) \n", + " eval_accuracy += tmp_eval_accuracy\n", + " nb_eval_steps += 1\n", + " print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n", + "\n", + "# plot training performance\n", + "plt.figure(figsize=(15,8))\n", + "plt.title(\"Training loss\")\n", + "plt.xlabel(\"Batch\")\n", + "plt.ylabel(\"Loss\")\n", + "plt.plot(train_loss_set)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iXXp4l9lenRu" + }, + "source": [ + "The model performs very bad in terms of accuracy. Pre-processing the data and hyper parameter tuning will help us to achieve better results." + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "collapsed_sections": [], + "name": "01_BioBERT_Demo.ipynb", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.3" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/Ch10/02_LexNLP.ipynb b/Ch10/02_LexNLP.ipynb index cd700b7..c4b403c 100644 --- a/Ch10/02_LexNLP.ipynb +++ b/Ch10/02_LexNLP.ipynb @@ -20,199 +20,7 @@ "id": "9N42o6gmX4mt", "outputId": "f5acd480-164a-430c-9d7c-3e40867e96e6" }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[?25hCollecting scipy==1.0.0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d8/5e/caa01ba7be11600b6a9d39265440d7b3be3d69206da887c42bef049521f2/scipy-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (50.0MB)\n", - "\u001b[K |████████████████████████████████| 50.0MB 74kB/s \n", - "\u001b[?25hCollecting gensim==3.4.0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/33/33/df6cb7acdcec5677ed130f4800f67509d24dbec74a03c329fcbf6b0864f0/gensim-3.4.0-cp36-cp36m-manylinux1_x86_64.whl (22.6MB)\n", - "\u001b[K |████████████████████████████████| 22.6MB 10.7MB/s \n", - "\u001b[?25hCollecting typing==3.6.2\n", - " Downloading https://files.pythonhosted.org/packages/44/88/d09c6a7fe1af4a02f16d2f1766212bec752aadb04e5699a9706a10a1a37d/typing-3.6.2-py3-none-any.whl\n", - "Collecting us==1.0.0\n", - " Downloading https://files.pythonhosted.org/packages/72/83/8731cbf5afcf3434c0b24cfc520c11fd27bfc8a6878114662f4e3dbdab71/us-1.0.0.tar.gz\n", - "Collecting Unidecode==0.4.21\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/01/a1/9d7f3138ee3d79a1ab865a2cb38200ca778d85121db19fe264c76c981184/Unidecode-0.04.21-py2.py3-none-any.whl (228kB)\n", - "\u001b[K |████████████████████████████████| 235kB 56.6MB/s \n", - "\u001b[?25hCollecting reporters-db==1.0.12.1\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/0d/6c/16c7c3849a25d2c3af5ef6e05d768d8e86e74aa9051df4728325c5f31f46/reporters_db-1.0.12.1-py2.py3-none-any.whl (60kB)\n", - "\u001b[K |████████████████████████████████| 61kB 8.7MB/s \n", - "\u001b[?25hCollecting scikit-learn==0.21.3\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/a0/c5/d2238762d780dde84a20b8c761f563fe882b88c5a5fb03c056547c442a19/scikit_learn-0.21.3-cp36-cp36m-manylinux1_x86_64.whl (6.7MB)\n", - "\u001b[K |████████████████████████████████| 6.7MB 51.5MB/s \n", - "\u001b[?25hRequirement already satisfied: python-dateutil>=2.4.2 in /usr/local/lib/python3.6/dist-packages (from datefinder-lexpredict==0.6.2->lexnlp) (2.6.1)\n", - "Requirement already satisfied: pytz in /usr/local/lib/python3.6/dist-packages (from datefinder-lexpredict==0.6.2->lexnlp) (2018.9)\n", - "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests==2.22.0->lexnlp) (2.8)\n", - "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests==2.22.0->lexnlp) (3.0.4)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests==2.22.0->lexnlp) (1.24.3)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests==2.22.0->lexnlp) (2019.11.28)\n", - "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk==3.4.5->lexnlp) (1.12.0)\n", - "Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from pandas==0.23.4->lexnlp) (1.17.5)\n", - "Requirement already satisfied: tzlocal in /usr/local/lib/python3.6/dist-packages (from dateparser==0.7.0->lexnlp) (1.5.1)\n", - "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.6/dist-packages (from gensim==3.4.0->lexnlp) (1.9.0)\n", - "Collecting jellyfish==0.5.6\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/94/48/ddb1458d966f0a84e472d059d87a9d1527df7768a725132fc1d810728386/jellyfish-0.5.6.tar.gz (132kB)\n", - "\u001b[K |████████████████████████████████| 133kB 59.3MB/s \n", - "\u001b[?25hRequirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn==0.21.3->lexnlp) (0.14.1)\n", - "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim==3.4.0->lexnlp) (1.11.15)\n", - "Requirement already satisfied: boto>=2.32 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim==3.4.0->lexnlp) (2.49.0)\n", - "Requirement already satisfied: botocore<1.15.0,>=1.14.15 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim==3.4.0->lexnlp) (1.14.15)\n", - "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim==3.4.0->lexnlp) (0.3.3)\n", - "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim==3.4.0->lexnlp) (0.9.4)\n", - "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.15.0,>=1.14.15->boto3->smart-open>=1.2.1->gensim==3.4.0->lexnlp) (0.15.2)\n", - "Building wheels for collected packages: nltk, num2words, regex, us, jellyfish\n", - " Building wheel for nltk (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for nltk: filename=nltk-3.4.5-cp36-none-any.whl size=1449905 sha256=6f66b493ebcdf90d2d5cd9cd9de5c82673122f94818739a1f53419c266d5eba9\n", - " Stored in directory: /root/.cache/pip/wheels/96/86/f6/68ab24c23f207c0077381a5e3904b2815136b879538a24b483\n", - " Building wheel for num2words (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for num2words: filename=num2words-0.5.7-cp36-none-any.whl size=84777 sha256=20060b7d3d1cb3384b877fc1e242344d95d683bb39253db9c27fa747927b0fae\n", - " Stored in directory: /root/.cache/pip/wheels/e5/f6/a1/78cb080e8529bfa0c6ee2461563a4ff96efc0b6fe89cb31457\n", - " Building wheel for regex (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for regex: filename=regex-2017.9.23-cp36-cp36m-linux_x86_64.whl size=538812 sha256=84302d2b22ab2cbf1a11efc8b01314807a51158cb4eb5a89d4781ed760f14ac0\n", - " Stored in directory: /root/.cache/pip/wheels/57/19/41/e7d239b4a53386fe9de49f9e4328799569bbeac8b8b3748876\n", - " Building wheel for us (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for us: filename=us-1.0.0-cp36-none-any.whl size=11833 sha256=e20c353277211701f7198ba7cf940d7b417c1f8dff30432704bb33c9edc95495\n", - " Stored in directory: /root/.cache/pip/wheels/b3/98/40/cb8be35d7779a0ae4372c84e7a585c947bfc41540fd8999e53\n", - " Building wheel for jellyfish (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for jellyfish: filename=jellyfish-0.5.6-cp36-cp36m-linux_x86_64.whl size=74535 sha256=5bfbbb44f6f26fa030b698cc316234cb6139300c870e593cee74378bec96c80c\n", - " Stored in directory: /root/.cache/pip/wheels/3b/29/06/8d686d24f742cb89e7bde7f26f18cb9e89b3c8bcd6999cb12a\n", - "Successfully built nltk num2words regex us jellyfish\n", - "\u001b[31mERROR: xarray 0.14.1 has requirement pandas>=0.24, but you'll have pandas 0.23.4 which is incompatible.\u001b[0m\n", - "\u001b[31mERROR: seaborn 0.10.0 has requirement scipy>=1.0.1, but you'll have scipy 1.0.0 which is incompatible.\u001b[0m\n", - "\u001b[31mERROR: plotnine 0.6.0 has requirement pandas>=0.25.0, but you'll have pandas 0.23.4 which is incompatible.\u001b[0m\n", - "\u001b[31mERROR: plotnine 0.6.0 has requirement scipy>=1.2.0, but you'll have scipy 1.0.0 which is incompatible.\u001b[0m\n", - "\u001b[31mERROR: mizani 0.6.0 has requirement pandas>=0.25.0, but you'll have pandas 0.23.4 which is incompatible.\u001b[0m\n", - "\u001b[31mERROR: google-colab 1.0.0 has requirement pandas~=0.25.0; python_version >= \"3.0\", but you'll have pandas 0.23.4 which is incompatible.\u001b[0m\n", - "\u001b[31mERROR: google-colab 1.0.0 has requirement requests~=2.21.0, but you'll have requests 2.22.0 which is incompatible.\u001b[0m\n", - "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n", - "\u001b[31mERROR: cvxpy 1.0.25 has requirement scipy>=1.1.0, but you'll have scipy 1.0.0 which is incompatible.\u001b[0m\n", - "\u001b[31mERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.\u001b[0m\n", - "Installing collected packages: regex, datefinder-lexpredict, requests, nltk, num2words, pandas, pycountry, dateparser, scipy, gensim, typing, jellyfish, us, Unidecode, reporters-db, scikit-learn, lexnlp\n", - " Found existing installation: regex 2019.12.20\n", - " Uninstalling regex-2019.12.20:\n", - " Successfully uninstalled regex-2019.12.20\n", - " Found existing installation: requests 2.21.0\n", - " Uninstalling requests-2.21.0:\n", - " Successfully uninstalled requests-2.21.0\n", - " Found existing installation: nltk 3.2.5\n", - " Uninstalling nltk-3.2.5:\n", - " Successfully uninstalled nltk-3.2.5\n", - " Found existing installation: pandas 0.25.3\n", - " Uninstalling pandas-0.25.3:\n", - " Successfully uninstalled pandas-0.25.3\n", - " Found existing installation: scipy 1.4.1\n", - " Uninstalling scipy-1.4.1:\n", - " Successfully uninstalled scipy-1.4.1\n", - " Found existing installation: gensim 3.6.0\n", - " Uninstalling gensim-3.6.0:\n", - " Successfully uninstalled gensim-3.6.0\n", - " Found existing installation: typing 3.6.6\n", - " Uninstalling typing-3.6.6:\n", - " Successfully uninstalled typing-3.6.6\n", - " Found existing installation: scikit-learn 0.22.1\n", - " Uninstalling scikit-learn-0.22.1:\n", - " Successfully uninstalled scikit-learn-0.22.1\n", - "Successfully installed Unidecode-0.4.21 datefinder-lexpredict-0.6.2 dateparser-0.7.0 gensim-3.4.0 jellyfish-0.5.6 lexnlp-1.4.0 nltk-3.4.5 num2words-0.5.7 pandas-0.23.4 pycountry-18.5.26 regex-2017.9.23 reporters-db-1.0.12.1 requests-2.22.0 scikit-learn-0.21.3 scipy-1.0.0 typing-3.6.2 us-1.0.0\n" - ] - }, - { - "data": { - "application/vnd.colab-display-data+json": { - "pip_warning": { - "packages": [ - "pandas", - "requests", - "typing" - ] - } - } - }, - "metadata": { - "tags": [] - }, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Collecting textract\n", - " Downloading https://files.pythonhosted.org/packages/32/31/ef9451e6e48a1a57e337c5f20d4ef58c1a13d91560d2574c738b1320bb8d/textract-1.6.3-py3-none-any.whl\n", - "Collecting argcomplete==1.10.0\n", - " Downloading https://files.pythonhosted.org/packages/4d/82/f44c9661e479207348a979b1f6f063625d11dc4ca6256af053719bbb0124/argcomplete-1.10.0-py2.py3-none-any.whl\n", - "Collecting EbookLib==0.17.1\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/00/38/7d6ab2e569a9165249619d73b7bc6be0e713a899a3bc2513814b6598a84c/EbookLib-0.17.1.tar.gz (111kB)\n", - "\u001b[K |████████████████████████████████| 112kB 8.4MB/s \n", - "\u001b[?25hCollecting pdfminer.six==20181108\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/8a/fd/6e8746e6965d1a7ea8e97253e3d79e625da5547e8f376f88de5d024bacb9/pdfminer.six-20181108-py2.py3-none-any.whl (5.6MB)\n", - "\u001b[K |████████████████████████████████| 5.6MB 50.4MB/s \n", - "\u001b[?25hCollecting docx2txt==0.8\n", - " Downloading https://files.pythonhosted.org/packages/7d/7d/60ee3f2b16d9bfdfa72e8599470a2c1a5b759cb113c6fe1006be28359327/docx2txt-0.8.tar.gz\n", - "Collecting python-pptx==0.6.18\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/bf/86/eb979f7b0333ec769041aae36df8b9f1bd8bea5bbad44620663890dce561/python-pptx-0.6.18.tar.gz (8.9MB)\n", - "\u001b[K |████████████████████████████████| 8.9MB 32.9MB/s \n", - "\u001b[?25hCollecting extract-msg==0.23.1\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/a1/90/84485a914ed90adb5e87df17e626be04162fbba146dfecf34643659a4633/extract_msg-0.23.1-py2.py3-none-any.whl (45kB)\n", - "\u001b[K |████████████████████████████████| 51kB 7.3MB/s \n", - "\u001b[?25hRequirement already satisfied: six==1.12.0 in /usr/local/lib/python3.6/dist-packages (from textract) (1.12.0)\n", - "Collecting xlrd==1.2.0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/b0/16/63576a1a001752e34bf8ea62e367997530dc553b689356b9879339cf45a4/xlrd-1.2.0-py2.py3-none-any.whl (103kB)\n", - "\u001b[K |████████████████████████████████| 112kB 59.9MB/s \n", - "\u001b[?25hCollecting beautifulsoup4==4.8.0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/1a/b7/34eec2fe5a49718944e215fde81288eec1fa04638aa3fb57c1c6cd0f98c3/beautifulsoup4-4.8.0-py3-none-any.whl (97kB)\n", - "\u001b[K |████████████████████████████████| 102kB 13.2MB/s \n", - "\u001b[?25hRequirement already satisfied: chardet==3.0.4 in /usr/local/lib/python3.6/dist-packages (from textract) (3.0.4)\n", - "Collecting SpeechRecognition==3.8.1\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/26/e1/7f5678cd94ec1234269d23756dbdaa4c8cfaed973412f88ae8adf7893a50/SpeechRecognition-3.8.1-py2.py3-none-any.whl (32.8MB)\n", - "\u001b[K |████████████████████████████████| 32.8MB 125kB/s \n", - "\u001b[?25hRequirement already satisfied: lxml in /usr/local/lib/python3.6/dist-packages (from EbookLib==0.17.1->textract) (4.2.6)\n", - "Requirement already satisfied: sortedcontainers in /usr/local/lib/python3.6/dist-packages (from pdfminer.six==20181108->textract) (2.1.0)\n", - "Collecting pycryptodome\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/af/16/da16a22d47bac9bf9db39f3b9af74e8eeed8855c0df96be20b580ef92fff/pycryptodome-3.9.7-cp36-cp36m-manylinux1_x86_64.whl (13.7MB)\n", - "\u001b[K |████████████████████████████████| 13.7MB 266kB/s \n", - "\u001b[?25hRequirement already satisfied: Pillow>=3.3.2 in /usr/local/lib/python3.6/dist-packages (from python-pptx==0.6.18->textract) (6.2.2)\n", - "Collecting XlsxWriter>=0.5.7\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/00/1f/2092a81056d36c1b6651a645aa84c1f76bcee03103072d4fe1cb58501d69/XlsxWriter-1.2.8-py2.py3-none-any.whl (141kB)\n", - "\u001b[K |████████████████████████████████| 143kB 49.4MB/s \n", - "\u001b[?25hCollecting olefile==0.46\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/34/81/e1ac43c6b45b4c5f8d9352396a14144bba52c8fec72a80f425f6a4d653ad/olefile-0.46.zip (112kB)\n", - "\u001b[K |████████████████████████████████| 112kB 50.9MB/s \n", - "\u001b[?25hRequirement already satisfied: tzlocal==1.5.1 in /usr/local/lib/python3.6/dist-packages (from extract-msg==0.23.1->textract) (1.5.1)\n", - "Collecting imapclient==2.1.0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/dc/39/e1c2c2c6e2356ab6ea81fcfc0a74b044b311d6a91a45300811d9a6077ef7/IMAPClient-2.1.0-py2.py3-none-any.whl (73kB)\n", - "\u001b[K |████████████████████████████████| 81kB 11.2MB/s \n", - "\u001b[?25hCollecting soupsieve>=1.2\n", - " Downloading https://files.pythonhosted.org/packages/05/cf/ea245e52f55823f19992447b008bcbb7f78efc5960d77f6c34b5b45b36dd/soupsieve-2.0-py2.py3-none-any.whl\n", - "Requirement already satisfied: pytz in /usr/local/lib/python3.6/dist-packages (from tzlocal==1.5.1->extract-msg==0.23.1->textract) (2018.9)\n", - "Building wheels for collected packages: EbookLib, docx2txt, python-pptx, olefile\n", - " Building wheel for EbookLib (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for EbookLib: filename=EbookLib-0.17.1-cp36-none-any.whl size=38164 sha256=200dbc555f17bfa257283a17bb6b4262380573c5df4db972956129e774b20fa7\n", - " Stored in directory: /root/.cache/pip/wheels/84/11/01/951369cbbf8f96878786a1f4da68bd7ac19a5d945b38e03d54\n", - " Building wheel for docx2txt (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for docx2txt: filename=docx2txt-0.8-cp36-none-any.whl size=3963 sha256=b8373bf45773da61c2bd49a7328b1c8e9c50f9beb95ed702e9f4f8f8ac59024a\n", - " Stored in directory: /root/.cache/pip/wheels/b2/1f/26/a051209bbb77fc6bcfae2bb7e01fa0ff941b82292ab084d596\n", - " Building wheel for python-pptx (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for python-pptx: filename=python_pptx-0.6.18-cp36-none-any.whl size=275706 sha256=1274975ede9f1d9ad5cb824ee2825f89e03e57e79054484146e5ac29a822f031\n", - " Stored in directory: /root/.cache/pip/wheels/1f/1f/2c/29acca422b420a0b5210bd2cd7e9669804520d602d2462f20b\n", - " Building wheel for olefile (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for olefile: filename=olefile-0.46-py2.py3-none-any.whl size=35417 sha256=332dbdcabd3564ae6439ddb24862371facca48bd516957e2a1f28bfed359250c\n", - " Stored in directory: /root/.cache/pip/wheels/4b/f4/11/bc4166107c27f07fd7bba707ffcb439619197638a1ac986df3\n", - "Successfully built EbookLib docx2txt python-pptx olefile\n", - "Installing collected packages: argcomplete, EbookLib, pycryptodome, pdfminer.six, docx2txt, XlsxWriter, python-pptx, olefile, imapclient, extract-msg, xlrd, soupsieve, beautifulsoup4, SpeechRecognition, textract\n", - " Found existing installation: xlrd 1.1.0\n", - " Uninstalling xlrd-1.1.0:\n", - " Successfully uninstalled xlrd-1.1.0\n", - " Found existing installation: beautifulsoup4 4.6.3\n", - " Uninstalling beautifulsoup4-4.6.3:\n", - " Successfully uninstalled beautifulsoup4-4.6.3\n", - "Successfully installed EbookLib-0.17.1 SpeechRecognition-3.8.1 XlsxWriter-1.2.8 argcomplete-1.10.0 beautifulsoup4-4.8.0 docx2txt-0.8 extract-msg-0.23.1 imapclient-2.1.0 olefile-0.46 pdfminer.six-20181108 pycryptodome-3.9.7 python-pptx-0.6.18 soupsieve-2.0 textract-1.6.3 xlrd-1.2.0\n" - ] - } - ], + "outputs": [], "source": [ "!pip3 install lexnlp\n", "!pip3 install textract" @@ -220,7 +28,7 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 2, "metadata": { "colab": {}, "colab_type": "code", @@ -229,15 +37,14 @@ "outputs": [], "source": [ "#Importing the necessary packages\n", - "from google.colab import files\n", "import textract\n", "from pprint import pprint\n", - "import pandas as pd\n" + "import pandas as pd" ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -262,41 +69,25 @@ "outputId": "5a92163b-873d-433d-b4b5-bd3db0fcaeea" }, "outputs": [ - { - "data": { - "text/html": [ - "\n", - " \n", - " \n", - " Upload widget is only available when the cell has been executed in the\n", - " current browser session. Please rerun this cell to enable.\n", - " \n", - " " - ], - "text/plain": [ - "" - ] - }, - "metadata": { - "tags": [] - }, - "output_type": "display_data" - }, { "name": "stdout", "output_type": "stream", "text": [ - "Saving test - Copy.docx to test - Copy.docx\n" + "Not using Colab\n" ] } ], "source": [ - "test_upload = files.upload()" + "try:\n", + " from google.colab import files\n", + " test_upload = files.upload()\n", + "except ModuleNotFoundError:\n", + " print('Not using Colab')" ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -313,16 +104,19 @@ "str" ] }, - "execution_count": 3, - "metadata": { - "tags": [] - }, + "execution_count": 4, + "metadata": {}, "output_type": "execute_result" } ], "source": [ "#Convert .docx to .txt\n", - "text = textract.process('test - Copy.docx').decode('utf-8')\n", + "try:\n", + " import google.colab\n", + " text = textract.process('test - Copy.docx').decode('utf-8')\n", + "except ModuleNotFoundError:\n", + " text = textract.process('Data/test - Copy.docx').decode('utf-8')\n", + "\n", "type(text)" ] }, @@ -335,7 +129,7 @@ }, { "cell_type": "code", - "execution_count": 38, + "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -456,21 +250,33 @@ "" ], "text/plain": [ - " location_start ... Act_annotations\n", - "0 6233 ... [act] at (6233..6264), loc: en\n", - "1 6346 ... [act] at (6346..6377), loc: en\n", - "2 9158 ... [act] at (9158..9176), loc: en\n", - "3 15403 ... [act] at (15403..15419), loc: en\n", - "4 15691 ... [act] at (15691..15707), loc: en\n", - "5 15806 ... [act] at (15806..15821), loc: en\n", + " location_start location_end act_name section year \\\n", + "0 6233 6264 Securities Exchange Act 1934 \n", + "1 6346 6377 Securities Exchange Act 1934 \n", + "2 9158 9176 Securities Act \n", + "3 15403 15419 Securities Act \n", + "4 15691 15707 Securities Act \n", + "5 15806 15821 Securities Act \n", + "\n", + " ambiguous value \\\n", + "0 False Securities Exchange Act of 1934 \n", + "1 False Securities Exchange Act of 1934 \n", + "2 False Securities Act.\\n\\n“ \n", + "3 False Securities Act, \n", + "4 False Securities Act, \n", + "5 False Securities Act \n", "\n", - "[6 rows x 8 columns]" + " Act_annotations \n", + "0 [act] at (6233..6264), loc: en \n", + "1 [act] at (6346..6377), loc: en \n", + "2 [act] at (9158..9176), loc: en \n", + "3 [act] at (15403..15419), loc: en \n", + "4 [act] at (15691..15707), loc: en \n", + "5 [act] at (15806..15821), loc: en " ] }, - "execution_count": 38, - "metadata": { - "tags": [] - }, + "execution_count": 5, + "metadata": {}, "output_type": "execute_result" } ], @@ -498,7 +304,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -509,20 +315,6 @@ "outputId": "f6b58400-123b-45a4-9356-2493dd262e7b" }, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package punkt to /root/nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] /root/nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package wordnet to /root/nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n" - ] - }, { "name": "stdout", "output_type": "stream", @@ -559,136 +351,56 @@ " \n", " \n", " 0\n", - " 1933.0\n", - " [amount] at (113..119), loc: en\n", - " \n", - " \n", - " 1\n", - " 1000000.0\n", - " [amount] at (604..612), loc: en\n", - " \n", - " \n", - " 2\n", - " 19.0\n", - " [amount] at (648..651), loc: en\n", - " \n", - " \n", - " 3\n", - " 2.0\n", - " [amount] at (651..654), loc: en\n", - " \n", - " \n", - " 4\n", - " 2022.0\n", - " [amount] at (654..660), loc: en\n", - " \n", - " \n", - " 5\n", " 1.0\n", " [amount] at (829..834), loc: en\n", " \n", " \n", - " 6\n", + " 1\n", " 1.0\n", " [amount] at (944..949), loc: en\n", " \n", " \n", - " 7\n", - " 100000.0\n", - " [amount] at (1054..1062), loc: en\n", - " \n", - " \n", - " 8\n", - " 1.0\n", - " [amount] at (1114..1116), loc: en\n", - " \n", - " \n", - " 9\n", + " 2\n", " 1.0\n", - " [amount] at (1322..1324), loc: en\n", + " [amount] at (9029..9036), loc: en\n", " \n", " \n", - " 10\n", - " 2.0\n", - " [amount] at (1478..1480), loc: en\n", - " \n", - " \n", - " 11\n", + " 3\n", " 1.0\n", - " [amount] at (3855..3857), loc: en\n", + " [amount] at (16957..16962), loc: en\n", " \n", " \n", - " 12\n", + " 4\n", " 1.0\n", - " [amount] at (5791..5793), loc: en\n", + " [amount] at (19358..19363), loc: en\n", " \n", " \n", - " 13\n", + " 5\n", " 1.0\n", - " [amount] at (5896..5898), loc: en\n", + " [amount] at (19668..19673), loc: en\n", " \n", " \n", - " 14\n", + " 6\n", " 1.0\n", - " [amount] at (5912..5914), loc: en\n", - " \n", - " \n", - " 15\n", - " 2.0\n", - " [amount] at (5919..5921), loc: en\n", - " \n", - " \n", - " 16\n", - " 13.0\n", - " [amount] at (6210..6213), loc: en\n", - " \n", - " \n", - " 17\n", - " 14.0\n", - " [amount] at (6220..6223), loc: en\n", - " \n", - " \n", - " 18\n", - " 1934.0\n", - " [amount] at (6260..6266), loc: en\n", - " \n", - " \n", - " 19\n", - " 3.0\n", - " [amount] at (6334..6336), loc: en\n", + " [amount] at (19839..19844), loc: en\n", " \n", " \n", "\n", "" ], "text/plain": [ - " Amount Location in Contract\n", - "0 1933.0 [amount] at (113..119), loc: en\n", - "1 1000000.0 [amount] at (604..612), loc: en\n", - "2 19.0 [amount] at (648..651), loc: en\n", - "3 2.0 [amount] at (651..654), loc: en\n", - "4 2022.0 [amount] at (654..660), loc: en\n", - "5 1.0 [amount] at (829..834), loc: en\n", - "6 1.0 [amount] at (944..949), loc: en\n", - "7 100000.0 [amount] at (1054..1062), loc: en\n", - "8 1.0 [amount] at (1114..1116), loc: en\n", - "9 1.0 [amount] at (1322..1324), loc: en\n", - "10 2.0 [amount] at (1478..1480), loc: en\n", - "11 1.0 [amount] at (3855..3857), loc: en\n", - "12 1.0 [amount] at (5791..5793), loc: en\n", - "13 1.0 [amount] at (5896..5898), loc: en\n", - "14 1.0 [amount] at (5912..5914), loc: en\n", - "15 2.0 [amount] at (5919..5921), loc: en\n", - "16 13.0 [amount] at (6210..6213), loc: en\n", - "17 14.0 [amount] at (6220..6223), loc: en\n", - "18 1934.0 [amount] at (6260..6266), loc: en\n", - "19 3.0 [amount] at (6334..6336), loc: en" + " Amount Location in Contract\n", + "0 1.0 [amount] at (829..834), loc: en\n", + "1 1.0 [amount] at (944..949), loc: en\n", + "2 1.0 [amount] at (9029..9036), loc: en\n", + "3 1.0 [amount] at (16957..16962), loc: en\n", + "4 1.0 [amount] at (19358..19363), loc: en\n", + "5 1.0 [amount] at (19668..19673), loc: en\n", + "6 1.0 [amount] at (19839..19844), loc: en" ] }, - "execution_count": 32, - "metadata": { - "tags": [] - }, + "execution_count": 6, + "metadata": {}, "output_type": "execute_result" } ], @@ -712,7 +424,7 @@ "\n", "\n", "df = pd.DataFrame(data=data_amts,columns=['Amount','Location in Contract'])\n", - "df.head(20)\n" + "df.head(20)" ] }, { @@ -724,7 +436,7 @@ }, { "cell_type": "code", - "execution_count": 39, + "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -739,7 +451,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "Dates in the Contract\n" + "Dates in the Contract\n", + "t 19-02-2022 2021-01-01 00:00:00 \n" ] }, { @@ -780,10 +493,8 @@ "0 2022-02-19" ] }, - "execution_count": 39, - "metadata": { - "tags": [] - }, + "execution_count": 7, + "metadata": {}, "output_type": "execute_result" } ], @@ -793,7 +504,7 @@ "\n", "data_dates = list(lexnlp.extract.en.dates.get_dates(text))\n", "df = pd.DataFrame(data=data_dates,columns=[\"Date\"])\n", - "df.head()\n" + "df.head()" ] }, { @@ -805,7 +516,7 @@ }, { "cell_type": "code", - "execution_count": 40, + "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -849,36 +560,18 @@ " \n", " \n", " \n", - " \n", - " 0\n", - " 1000000.0\n", - " USD\n", - " \n", - " \n", - " 1\n", - " 100000.0\n", - " USD\n", - " \n", - " \n", - " 2\n", - " 2.0\n", - " USD\n", - " \n", " \n", "\n", "" ], "text/plain": [ - " Amount Currency\n", - "0 1000000.0 USD\n", - "1 100000.0 USD\n", - "2 2.0 USD" + "Empty DataFrame\n", + "Columns: [Amount, Currency]\n", + "Index: []" ] }, - "execution_count": 40, - "metadata": { - "tags": [] - }, + "execution_count": 8, + "metadata": {}, "output_type": "execute_result" } ], @@ -888,7 +581,7 @@ "\n", "data_money = list(lexnlp.extract.en.money.get_money(text))\n", "df = pd.DataFrame(data=data_money,columns=[\"Amount\",\"Currency\"])\n", - "df.head()\n" + "df.head()" ] }, { @@ -900,7 +593,7 @@ }, { "cell_type": "code", - "execution_count": 45, + "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -953,9 +646,7 @@ "Index: []" ] }, - "metadata": { - "tags": [] - }, + "metadata": {}, "output_type": "display_data" }, { @@ -985,7 +676,7 @@ }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1034,11 +725,11 @@ " \n", " \n", " 1\n", - " Purchase Amount\n", + " Investor\n", " \n", " \n", " 2\n", - " Investor\n", + " Purchase Amount\n", " \n", " \n", " 3\n", @@ -1115,8 +806,8 @@ "text/plain": [ " Acts\n", "0 SECURITIES ACT\n", - "1 Purchase Amount\n", - "2 Investor\n", + "1 Investor\n", + "2 Purchase Amount\n", "3 Cash-Out Amount\n", "4 Conversion Amount\n", "5 Capital Stock\n", @@ -1136,10 +827,8 @@ "19 Safe Price" ] }, - "execution_count": 46, - "metadata": { - "tags": [] - }, + "execution_count": 10, + "metadata": {}, "output_type": "execute_result" } ], @@ -1161,7 +850,7 @@ }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1219,10 +908,8 @@ "0 ((c), None, Dissolution Event) Dissolution Event, (3862, 3883)" ] }, - "execution_count": 49, - "metadata": { - "tags": [] - }, + "execution_count": 11, + "metadata": {}, "output_type": "execute_result" } ], @@ -1247,7 +934,7 @@ }, { "cell_type": "code", - "execution_count": 50, + "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1291,24 +978,18 @@ " \n", " \n", " \n", - " \n", - " 0\n", - " (hour, 48.0, 2.0)\n", - " [duration] at (17772..17781), loc: en\n", - " \n", " \n", "\n", "" ], "text/plain": [ - " Durations Duration Annotation\n", - "0 (hour, 48.0, 2.0) [duration] at (17772..17781), loc: en" + "Empty DataFrame\n", + "Columns: [Durations, Duration Annotation]\n", + "Index: []" ] }, - "execution_count": 50, - "metadata": { - "tags": [] - }, + "execution_count": 12, + "metadata": {}, "output_type": "execute_result" } ], @@ -1321,7 +1002,7 @@ "data_dur_ann = list(lexnlp.extract.en.durations.get_duration_annotations_list(text))\n", "\n", "df = pd.DataFrame(data=list(zip(data_dur,data_dur_ann)),columns=['Durations','Duration Annotation'])\n", - "df.head()\n" + "df.head()" ] }, { @@ -1334,7 +1015,7 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1371,7 +1052,7 @@ "print(\"\")\n", "# print(l/]ist(lexnlp.nlp.en.tokens.get_nouns(text)))\n", "print(\"Lematization\")\n", - "print(list(lexnlp.nlp.en.tokens.get_verbs(text, lemmatize=True))[:10])\n" + "print(list(lexnlp.nlp.en.tokens.get_verbs(text, lemmatize=True))[:10])" ] }, { @@ -1383,7 +1064,7 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1819,7 +1500,7 @@ }, { "cell_type": "code", - "execution_count": 53, + "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2485,7 +2166,13 @@ " ('a', 'reincorporation'): 1,\n", " ('a', 'result'): 1,\n", " ('a', 'sale'): 1,\n", - " ('a', 'similar'): 1,\n", + " ('a', 'similar'): 1,\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ " ('a', 'space'): 1,\n", " ('a', 'tax-free'): 2,\n", " ('a', 'total'): 1,\n", @@ -3226,7 +2913,13 @@ " ('knowledge', 'and'): 1,\n", " ('lacking', 'one'): 1,\n", " ('latter', 'case'): 1,\n", - " ('law', 'or'): 1,\n", + " ('law', 'or'): 1,\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ " ('law', 'provisions'): 1,\n", " ('laws', ';'): 1,\n", " ('laws', 'and'): 1,\n", @@ -3906,7 +3599,13 @@ " ('the', 'basis'): 1,\n", " ('the', 'benefit'): 1,\n", " ('the', 'calculation'): 1,\n", - " ('the', 'capital'): 1,\n", + " ('the', 'capital'): 1,\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ " ('the', 'case'): 1,\n", " ('the', 'cash'): 1,\n", " ('the', 'closing'): 1,\n", @@ -4202,19 +3901,8 @@ "import lexnlp.nlp.en.transforms.tokens\n", "# print(dir(lexnlp.nlp.en.transforms.tokens))\n", "\n", - "pprint(dict(lexnlp.nlp.en.transforms.tokens.get_bigram_distribution(text)))\n" + "pprint(dict(lexnlp.nlp.en.transforms.tokens.get_bigram_distribution(text)))" ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "RgNP6BMpmx4x" - }, - "outputs": [], - "source": [] } ], "metadata": { @@ -4238,7 +3926,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.10" + "version": "3.8.3" } }, "nbformat": 4, diff --git a/Ch10/03_FinBERT.ipynb b/Ch10/03_FinBERT.ipynb index 0c004b0..3e82fe6 100644 --- a/Ch10/03_FinBERT.ipynb +++ b/Ch10/03_FinBERT.ipynb @@ -1,1551 +1,1610 @@ { - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "name": "FinBERT.ipynb", - "provenance": [], - "collapsed_sections": [] - }, - "kernelspec": { - "name": "python3", - "display_name": "Python 3" - }, - "accelerator": "GPU" + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "DB7J6qSNl-ds" + }, + "source": [ + "In this notebook we demonstrate how to leverage [FinBert](https://github.com/ProsusAI/finBERT) to classify the sentiment of a stocks.
You can download pre-trained FinBERT from [here](https://drive.google.com/drive/folders/1rcRXZhb3JLY3A_kIO8gMk8jacRyR-Ik6)" + ] }, - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "DB7J6qSNl-ds", - "colab_type": "text" - }, - "source": [ - "In this notebook we demonstrate how to leverage [FinBert](https://github.com/ProsusAI/finBERT) to classify the sentiment of a stocks.
You can download pre-trained FinBERT from [here](https://drive.google.com/drive/folders/1rcRXZhb3JLY3A_kIO8gMk8jacRyR-Ik6)" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "0exgvfpbjDpM", - "colab_type": "code", - "colab": {} - }, - "source": [ - "# https://drive.google.com/file/d/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL/view?usp=sharing \n", - "# https://drive.google.com/file/d/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN/view?usp=sharing" - ], - "execution_count": 0, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "m2bGe8FImiAB", - "colab_type": "text" - }, - "source": [ - "Huge shoutout to this [article](https://medium.com/@acpanjan/download-google-drive-files-using-wget-3c2c025a8b99) which helped in using wget to download the pre-trained model.
\n", - "To Download any version of FinBERT follow these steps:\n", - "1. Open the drive link and open the folder of the model you want to download\n", - "2. Right click on a file in the folder and select \"get sharable link\" option\n", - "3. Copy the part after 'd/' in the sharable URL\n", - "4. Replace the ID youve obtained intead of the id in the wget url if its different.\n", - "5. Repeat the process for the other file in the folder
\n", - "`!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=FILEID\" -O pytorch_model.bin && rm -rf /tmp/cookies.txt `" - ] + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "0exgvfpbjDpM" + }, + "outputs": [], + "source": [ + "# https://drive.google.com/file/d/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL/view?usp=sharing \n", + "# https://drive.google.com/file/d/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN/view?usp=sharing" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "m2bGe8FImiAB" + }, + "source": [ + "Huge shoutout to this [article](https://medium.com/@acpanjan/download-google-drive-files-using-wget-3c2c025a8b99) which helped in using wget to download the pre-trained model.
\n", + "To Download any version of FinBERT follow these steps:\n", + "1. Open the drive link and open the folder of the model you want to download\n", + "2. Right click on a file in the folder and select \"get sharable link\" option\n", + "3. Copy the part after 'd/' in the sharable URL\n", + "4. Replace the ID youve obtained intead of the id in the wget url if its different.\n", + "5. Repeat the process for the other file in the folder
\n", + "`!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=FILEID\" -O pytorch_model.bin && rm -rf /tmp/cookies.txt `" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "id": "rYwTnQmE8xeE" + }, + "outputs": [], + "source": [ + "import requests\n", + "\n", + "def download_file_from_google_drive(id, destination):\n", + " URL = \"https://docs.google.com/uc?export=download\"\n", + "\n", + " session = requests.Session()\n", + "\n", + " response = session.get(URL, params = { 'id' : id }, stream = True)\n", + " token = get_confirm_token(response)\n", + "\n", + " if token:\n", + " params = { 'id' : id, 'confirm' : token }\n", + " response = session.get(URL, params = params, stream = True)\n", + "\n", + " save_response_content(response, destination) \n", + "\n", + "def get_confirm_token(response):\n", + " for key, value in response.cookies.items():\n", + " if key.startswith('download_warning'):\n", + " return value\n", + "\n", + " return None\n", + "\n", + "def save_response_content(response, destination):\n", + " CHUNK_SIZE = 32768\n", + "\n", + " with open(destination, \"wb\") as f:\n", + " for chunk in response.iter_content(CHUNK_SIZE):\n", + " if chunk: # filter out keep-alive new chunks\n", + " f.write(chunk)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "oeEsKS51naF3" + }, + "source": [ + "### Model Weights" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "PBE63efS8xeF", + "outputId": "50e9d2fd-9ce1-43b7-f126-b7d14bfceb76" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "oeEsKS51naF3", - "colab_type": "text" - }, - "source": [ - "### Model Weights" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "--2021-06-18 11:56:04-- https://docs.google.com/uc?export=download&confirm=lcvH&id=1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL\n", + "Resolving docs.google.com (docs.google.com)... 172.217.13.78, 2607:f8b0:4004:808::200e\n", + "Connecting to docs.google.com (docs.google.com)|172.217.13.78|:443... connected.\n", + "HTTP request sent, awaiting response... 302 Moved Temporarily\n", + "Location: https://doc-0c-14-docs.googleusercontent.com/docs/securesc/f5es3p7t972v8dk9q7infe1bujplqemt/uvjon2ocb6me7e6ub76c5imbhk338srn/1624017300000/03819273116678412817/12187802221879198422Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e=download [following]\n", + "--2021-06-18 11:56:05-- https://doc-0c-14-docs.googleusercontent.com/docs/securesc/f5es3p7t972v8dk9q7infe1bujplqemt/uvjon2ocb6me7e6ub76c5imbhk338srn/1624017300000/03819273116678412817/12187802221879198422Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e=download\n", + "Resolving doc-0c-14-docs.googleusercontent.com (doc-0c-14-docs.googleusercontent.com)... 172.217.2.97, 2607:f8b0:4004:80a::2001\n", + "Connecting to doc-0c-14-docs.googleusercontent.com (doc-0c-14-docs.googleusercontent.com)|172.217.2.97|:443... connected.\n", + "HTTP request sent, awaiting response... 302 Found\n", + "Location: https://docs.google.com/nonceSigner?nonce=efq2i14bh7lr0&continue=https://doc-0c-14-docs.googleusercontent.com/docs/securesc/f5es3p7t972v8dk9q7infe1bujplqemt/uvjon2ocb6me7e6ub76c5imbhk338srn/1624017300000/03819273116678412817/12187802221879198422Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e%3Ddownload&hash=m2ofpqlr0v4trjqmbhch7b3bb4qui2k4 [following]\n", + "--2021-06-18 11:56:05-- https://docs.google.com/nonceSigner?nonce=efq2i14bh7lr0&continue=https://doc-0c-14-docs.googleusercontent.com/docs/securesc/f5es3p7t972v8dk9q7infe1bujplqemt/uvjon2ocb6me7e6ub76c5imbhk338srn/1624017300000/03819273116678412817/12187802221879198422Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e%3Ddownload&hash=m2ofpqlr0v4trjqmbhch7b3bb4qui2k4\n", + "Connecting to docs.google.com (docs.google.com)|172.217.13.78|:443... connected.\n", + "HTTP request sent, awaiting response... 302 Found\n", + "Location: https://doc-0c-14-docs.googleusercontent.com/docs/securesc/f5es3p7t972v8dk9q7infe1bujplqemt/uvjon2ocb6me7e6ub76c5imbhk338srn/1624017300000/03819273116678412817/12187802221879198422Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e=download&nonce=efq2i14bh7lr0&user=12187802221879198422Z&hash=vibjbelgvlm0ltfs66m77feo4tsfhoov [following]\n", + "--2021-06-18 11:56:05-- https://doc-0c-14-docs.googleusercontent.com/docs/securesc/f5es3p7t972v8dk9q7infe1bujplqemt/uvjon2ocb6me7e6ub76c5imbhk338srn/1624017300000/03819273116678412817/12187802221879198422Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e=download&nonce=efq2i14bh7lr0&user=12187802221879198422Z&hash=vibjbelgvlm0ltfs66m77feo4tsfhoov\n", + "Connecting to doc-0c-14-docs.googleusercontent.com (doc-0c-14-docs.googleusercontent.com)|172.217.2.97|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: unspecified [application/octet-stream]\n", + "Saving to: ‘pytorch_model.bin’\n", + "\n", + "pytorch_model.bin [ <=> ] 420.07M 186MB/s in 2.3s \n", + "\n", + "2021-06-18 11:56:07 (186 MB/s) - ‘pytorch_model.bin’ saved [440471924]\n", + "\n" + ] + } + ], + "source": [ + "try:\n", + " import google.colab\n", + " !wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL\" -O pytorch_model.bin && rm -rf /tmp/cookies.txt\n", + "except ModuleNotFoundError:\n", + " file_id = '1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL'\n", + " destination = 'Data/pytorch_model.bin'\n", + " download_file_from_google_drive(file_id, destination)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "M8loE2s7ncY7" + }, + "source": [ + "# Config.json" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "aLcZJCOJ8xeG", + "outputId": "8edb8e54-35b8-4597-b611-b2258561fc1c" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "5xvgKUNw4Ynj", - "colab_type": "code", - "outputId": "b6e7ccb7-2a6d-4152-a2e1-7735ce7322be", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 474 - } - }, - "source": [ - "!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL\" -O pytorch_model.bin && rm -rf /tmp/cookies.txt" - ], - "execution_count": 3, - "outputs": [ - { - "output_type": "stream", - "text": [ - "--2020-06-03 13:46:51-- https://docs.google.com/uc?export=download&confirm=mTWR&id=1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL\n", - "Resolving docs.google.com (docs.google.com)... 172.217.203.100, 172.217.203.138, 172.217.203.101, ...\n", - "Connecting to docs.google.com (docs.google.com)|172.217.203.100|:443... connected.\n", - "HTTP request sent, awaiting response... 302 Moved Temporarily\n", - "Location: https://doc-0o-1s-docs.googleusercontent.com/docs/securesc/p2jgqfea5lpvi9t30a0mh5qee7jl6ul4/71vece8u1gc2o64ve0vkbil306fgp7qv/1591191975000/03819273116678412817/07892353702974641314Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e=download [following]\n", - "--2020-06-03 13:46:51-- https://doc-0o-1s-docs.googleusercontent.com/docs/securesc/p2jgqfea5lpvi9t30a0mh5qee7jl6ul4/71vece8u1gc2o64ve0vkbil306fgp7qv/1591191975000/03819273116678412817/07892353702974641314Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e=download\n", - "Resolving doc-0o-1s-docs.googleusercontent.com (doc-0o-1s-docs.googleusercontent.com)... 173.194.217.132, 2607:f8b0:400c:c13::84\n", - "Connecting to doc-0o-1s-docs.googleusercontent.com (doc-0o-1s-docs.googleusercontent.com)|173.194.217.132|:443... connected.\n", - "HTTP request sent, awaiting response... 302 Found\n", - "Location: https://docs.google.com/nonceSigner?nonce=0jhvaoldr7lre&continue=https://doc-0o-1s-docs.googleusercontent.com/docs/securesc/p2jgqfea5lpvi9t30a0mh5qee7jl6ul4/71vece8u1gc2o64ve0vkbil306fgp7qv/1591191975000/03819273116678412817/07892353702974641314Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e%3Ddownload&hash=3666rugp3rbvhhg76ntm064276kvrss6 [following]\n", - "--2020-06-03 13:46:51-- https://docs.google.com/nonceSigner?nonce=0jhvaoldr7lre&continue=https://doc-0o-1s-docs.googleusercontent.com/docs/securesc/p2jgqfea5lpvi9t30a0mh5qee7jl6ul4/71vece8u1gc2o64ve0vkbil306fgp7qv/1591191975000/03819273116678412817/07892353702974641314Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e%3Ddownload&hash=3666rugp3rbvhhg76ntm064276kvrss6\n", - "Connecting to docs.google.com (docs.google.com)|172.217.203.100|:443... connected.\n", - "HTTP request sent, awaiting response... 302 Found\n", - "Location: https://doc-0o-1s-docs.googleusercontent.com/docs/securesc/p2jgqfea5lpvi9t30a0mh5qee7jl6ul4/71vece8u1gc2o64ve0vkbil306fgp7qv/1591191975000/03819273116678412817/07892353702974641314Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e=download&nonce=0jhvaoldr7lre&user=07892353702974641314Z&hash=6af9d481pfir4ok6aa5opn9lsbbf4tfq [following]\n", - "--2020-06-03 13:46:51-- https://doc-0o-1s-docs.googleusercontent.com/docs/securesc/p2jgqfea5lpvi9t30a0mh5qee7jl6ul4/71vece8u1gc2o64ve0vkbil306fgp7qv/1591191975000/03819273116678412817/07892353702974641314Z/1ycRUIpll2tfooWRvsinNzM_6B5tz3EtL?e=download&nonce=0jhvaoldr7lre&user=07892353702974641314Z&hash=6af9d481pfir4ok6aa5opn9lsbbf4tfq\n", - "Connecting to doc-0o-1s-docs.googleusercontent.com (doc-0o-1s-docs.googleusercontent.com)|173.194.217.132|:443... connected.\n", - "HTTP request sent, awaiting response... 200 OK\n", - "Length: unspecified [application/octet-stream]\n", - "Saving to: ‘pytorch_model.bin’\n", - "\n", - "pytorch_model.bin [ <=>] 420.07M 59.1MB/s in 7.1s \n", - "\n", - "2020-06-03 13:46:58 (59.1 MB/s) - ‘pytorch_model.bin’ saved [440471924]\n", - "\n" - ], - "name": "stdout" - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "--2021-06-18 11:56:11-- https://docs.google.com/uc?export=download&confirm=&id=1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN\n", + "Resolving docs.google.com (docs.google.com)... 172.217.13.78, 2607:f8b0:4004:808::200e\n", + "Connecting to docs.google.com (docs.google.com)|172.217.13.78|:443... connected.\n", + "HTTP request sent, awaiting response... 302 Moved Temporarily\n", + "Location: https://doc-0o-3g-docs.googleusercontent.com/docs/securesc/89b0o114i83nnhmforufd4sjppd7grna/1e1eoef6b6nqakmb6hae0rarjg91snaf/1624017300000/03819273116678412817/03739935542666294484Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e=download [following]\n", + "--2021-06-18 11:56:12-- https://doc-0o-3g-docs.googleusercontent.com/docs/securesc/89b0o114i83nnhmforufd4sjppd7grna/1e1eoef6b6nqakmb6hae0rarjg91snaf/1624017300000/03819273116678412817/03739935542666294484Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e=download\n", + "Resolving doc-0o-3g-docs.googleusercontent.com (doc-0o-3g-docs.googleusercontent.com)... 172.217.2.97, 2607:f8b0:4004:80a::2001\n", + "Connecting to doc-0o-3g-docs.googleusercontent.com (doc-0o-3g-docs.googleusercontent.com)|172.217.2.97|:443... connected.\n", + "HTTP request sent, awaiting response... 302 Found\n", + "Location: https://docs.google.com/nonceSigner?nonce=foir13t68blmi&continue=https://doc-0o-3g-docs.googleusercontent.com/docs/securesc/89b0o114i83nnhmforufd4sjppd7grna/1e1eoef6b6nqakmb6hae0rarjg91snaf/1624017300000/03819273116678412817/03739935542666294484Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e%3Ddownload&hash=sun3tc9i9a1np6bnjl9hpdotslsbqhpf [following]\n", + "--2021-06-18 11:56:12-- https://docs.google.com/nonceSigner?nonce=foir13t68blmi&continue=https://doc-0o-3g-docs.googleusercontent.com/docs/securesc/89b0o114i83nnhmforufd4sjppd7grna/1e1eoef6b6nqakmb6hae0rarjg91snaf/1624017300000/03819273116678412817/03739935542666294484Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e%3Ddownload&hash=sun3tc9i9a1np6bnjl9hpdotslsbqhpf\n", + "Connecting to docs.google.com (docs.google.com)|172.217.13.78|:443... connected.\n", + "HTTP request sent, awaiting response... 302 Found\n", + "Location: https://doc-0o-3g-docs.googleusercontent.com/docs/securesc/89b0o114i83nnhmforufd4sjppd7grna/1e1eoef6b6nqakmb6hae0rarjg91snaf/1624017300000/03819273116678412817/03739935542666294484Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e=download&nonce=foir13t68blmi&user=03739935542666294484Z&hash=qs8dokot4l0tq62ehgd7s87lhij2rml0 [following]\n", + "--2021-06-18 11:56:12-- https://doc-0o-3g-docs.googleusercontent.com/docs/securesc/89b0o114i83nnhmforufd4sjppd7grna/1e1eoef6b6nqakmb6hae0rarjg91snaf/1624017300000/03819273116678412817/03739935542666294484Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e=download&nonce=foir13t68blmi&user=03739935542666294484Z&hash=qs8dokot4l0tq62ehgd7s87lhij2rml0\n", + "Connecting to doc-0o-3g-docs.googleusercontent.com (doc-0o-3g-docs.googleusercontent.com)|172.217.2.97|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 313 [application/json]\n", + "Saving to: ‘config.json’\n", + "\n", + "config.json 100%[===================>] 313 --.-KB/s in 0s \n", + "\n", + "2021-06-18 11:56:12 (20.5 MB/s) - ‘config.json’ saved [313/313]\n", + "\n" + ] + } + ], + "source": [ + "try:\n", + " import google.colab\n", + " !wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN\" -O config.json && rm -rf /tmp/cookies.txt\n", + "except ModuleNotFoundError:\n", + " file_id = '1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN'\n", + " destination = 'Data/bert_config.json'\n", + " download_file_from_google_drive(file_id, destination)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6rYRtaycnfcO" + }, + "source": [ + "#Vocabulary\n", + "Finbert drive link does not have a vocab.txt file. In the FinBERT repo they have mentioned that the same vocab file was used as for the 'bert-base-uncased' model. So we are going to download the vocab.txt of the 'bert-base-uncased' model" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "X77Z0RJY-DJY", + "outputId": "0061b41b-f98c-44eb-eeee-afbe8e730c09" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "M8loE2s7ncY7", - "colab_type": "text" - }, - "source": [ - "# Config.json" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: wget in /usr/local/lib/python3.7/dist-packages (3.2)\n" + ] + } + ], + "source": [ + "!pip install wget" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "id": "wc-sZkDP8xeH" + }, + "outputs": [], + "source": [ + "import wget\n", + "try :\n", + " import google.colab\n", + " wget.download(\"https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt\")\n", + "\n", + "except ModuleNotFoundError :\n", + " wget.download(\"https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt\", \"Data\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NmrwwAkToCz7" + }, + "source": [ + "### Move all the files into a FinBERT directory. This is the format that is required by the huggingface transformers library" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "hmjzyZEA48ZX" + }, + "outputs": [], + "source": [ + "try :\n", + " import google.colab \n", + " !mkdir finbert\n", + " !mv pytorch_model.bin finbert/\n", + " !mv config.json finbert/\n", + " !mv bert-base-uncased-vocab.txt vocab.txt\n", + " !mv vocab.txt finbert/\n", + " \n", + "except ModuleNotFoundError:\n", + " import os\n", + " import shutil\n", + " os.mkdir('Data/finbert')\n", + " shutil.move('Data/pytorch_model.bin' , 'Data/finbert')\n", + " shutil.move('Data/bert_config.json', 'Data/finbert')\n", + " os.rename('Data/bert-base-uncased-vocab.txt' , 'Data/vocab.txt')\n", + " shutil.move('Data/vocab.txt', 'Data/finbert')\n", + " \n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MUCJlVHHoNd7" + }, + "source": [ + "### Installing packages" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "qsMk4azI5pcD", + "outputId": "73b83a2e-96c8-4387-9713-44f87080cfe0" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "uMHmlDnY4d1X", - "colab_type": "code", - "outputId": "43b60bc0-7501-47fe-cdc1-d04a3b4ce103", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 474 - } - }, - "source": [ - "!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN\" -O config.json && rm -rf /tmp/cookies.txt" - ], - "execution_count": 4, - "outputs": [ - { - "output_type": "stream", - "text": [ - "--2020-06-03 13:47:15-- https://docs.google.com/uc?export=download&confirm=&id=1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN\n", - "Resolving docs.google.com (docs.google.com)... 172.217.203.100, 172.217.203.138, 172.217.203.101, ...\n", - "Connecting to docs.google.com (docs.google.com)|172.217.203.100|:443... connected.\n", - "HTTP request sent, awaiting response... 302 Moved Temporarily\n", - "Location: https://doc-10-1g-docs.googleusercontent.com/docs/securesc/kqkqjae01qsvjsmmsdoi1baho0ambc4g/u1gn55kprdm4m2ngfre0sngthsa4m7lu/1591191975000/03819273116678412817/06303696199905918493Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e=download [following]\n", - "--2020-06-03 13:47:15-- https://doc-10-1g-docs.googleusercontent.com/docs/securesc/kqkqjae01qsvjsmmsdoi1baho0ambc4g/u1gn55kprdm4m2ngfre0sngthsa4m7lu/1591191975000/03819273116678412817/06303696199905918493Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e=download\n", - "Resolving doc-10-1g-docs.googleusercontent.com (doc-10-1g-docs.googleusercontent.com)... 173.194.217.132, 2607:f8b0:400c:c13::84\n", - "Connecting to doc-10-1g-docs.googleusercontent.com (doc-10-1g-docs.googleusercontent.com)|173.194.217.132|:443... connected.\n", - "HTTP request sent, awaiting response... 302 Found\n", - "Location: https://docs.google.com/nonceSigner?nonce=n539eogb0437c&continue=https://doc-10-1g-docs.googleusercontent.com/docs/securesc/kqkqjae01qsvjsmmsdoi1baho0ambc4g/u1gn55kprdm4m2ngfre0sngthsa4m7lu/1591191975000/03819273116678412817/06303696199905918493Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e%3Ddownload&hash=b6u197v79bkq4eerh6pdma7nf2t3cdgg [following]\n", - "--2020-06-03 13:47:15-- https://docs.google.com/nonceSigner?nonce=n539eogb0437c&continue=https://doc-10-1g-docs.googleusercontent.com/docs/securesc/kqkqjae01qsvjsmmsdoi1baho0ambc4g/u1gn55kprdm4m2ngfre0sngthsa4m7lu/1591191975000/03819273116678412817/06303696199905918493Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e%3Ddownload&hash=b6u197v79bkq4eerh6pdma7nf2t3cdgg\n", - "Connecting to docs.google.com (docs.google.com)|172.217.203.100|:443... connected.\n", - "HTTP request sent, awaiting response... 302 Found\n", - "Location: https://doc-10-1g-docs.googleusercontent.com/docs/securesc/kqkqjae01qsvjsmmsdoi1baho0ambc4g/u1gn55kprdm4m2ngfre0sngthsa4m7lu/1591191975000/03819273116678412817/06303696199905918493Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e=download&nonce=n539eogb0437c&user=06303696199905918493Z&hash=r12snp2gdramkns0t2kmck5m9cu4laim [following]\n", - "--2020-06-03 13:47:15-- https://doc-10-1g-docs.googleusercontent.com/docs/securesc/kqkqjae01qsvjsmmsdoi1baho0ambc4g/u1gn55kprdm4m2ngfre0sngthsa4m7lu/1591191975000/03819273116678412817/06303696199905918493Z/1MJMEeVLBdQ79F1TUewP_WR0uFzR81UPN?e=download&nonce=n539eogb0437c&user=06303696199905918493Z&hash=r12snp2gdramkns0t2kmck5m9cu4laim\n", - "Connecting to doc-10-1g-docs.googleusercontent.com (doc-10-1g-docs.googleusercontent.com)|173.194.217.132|:443... connected.\n", - "HTTP request sent, awaiting response... 200 OK\n", - "Length: 313 [application/json]\n", - "Saving to: ‘config.json’\n", - "\n", - "\rconfig.json 0%[ ] 0 --.-KB/s \rconfig.json 100%[===================>] 313 --.-KB/s in 0s \n", - "\n", - "2020-06-03 13:47:15 (19.1 MB/s) - ‘config.json’ saved [313/313]\n", - "\n" - ], - "name": "stdout" - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting pytorch_transformers\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/a3/b7/d3d18008a67e0b968d1ab93ad444fc05699403fa662f634b2f2c318a508b/pytorch_transformers-1.2.0-py3-none-any.whl (176kB)\n", + "\u001b[K |████████████████████████████████| 184kB 25.7MB/s \n", + "\u001b[?25hCollecting boto3\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/39/96/5c009100d651f4646639f9777a82777430897590f5fd28bbc077cace09bf/boto3-1.17.97-py2.py3-none-any.whl (131kB)\n", + "\u001b[K |████████████████████████████████| 133kB 36.5MB/s \n", + "\u001b[?25hRequirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (2019.12.20)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (2.23.0)\n", + "Collecting sentencepiece\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/ac/aa/1437691b0c7c83086ebb79ce2da16e00bef024f24fec2a5161c35476f499/sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2MB)\n", + "\u001b[K |████████████████████████████████| 1.2MB 22.1MB/s \n", + "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (4.41.1)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (1.19.5)\n", + "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from pytorch_transformers) (1.8.1+cu101)\n", + "Collecting sacremoses\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/75/ee/67241dc87f266093c533a2d4d3d69438e57d7a90abb216fa076e7d475d4a/sacremoses-0.0.45-py3-none-any.whl (895kB)\n", + "\u001b[K |████████████████████████████████| 901kB 51.6MB/s \n", + "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n", + " Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n", + "Collecting s3transfer<0.5.0,>=0.4.0\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/63/d0/693477c688348654ddc21dcdce0817653a294aa43f41771084c25e7ff9c7/s3transfer-0.4.2-py2.py3-none-any.whl (79kB)\n", + "\u001b[K |████████████████████████████████| 81kB 11.1MB/s \n", + "\u001b[?25hCollecting botocore<1.21.0,>=1.20.97\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/94/a2/bd7cd40c61a71c3daede70158b5c0aca918057f63cfbef015c56cccdb6b2/botocore-1.20.97-py2.py3-none-any.whl (7.6MB)\n", + "\u001b[K |████████████████████████████████| 7.6MB 47.6MB/s \n", + "\u001b[?25hRequirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_transformers) (2021.5.30)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_transformers) (1.24.3)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_transformers) (3.0.4)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_transformers) (2.10)\n", + "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.0.0->pytorch_transformers) (3.7.4.3)\n", + "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->pytorch_transformers) (1.15.0)\n", + "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->pytorch_transformers) (1.0.1)\n", + "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->pytorch_transformers) (7.1.2)\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.21.0,>=1.20.97->boto3->pytorch_transformers) (2.8.1)\n", + "\u001b[31mERROR: botocore 1.20.97 has requirement urllib3<1.27,>=1.25.4, but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n", + "Installing collected packages: jmespath, botocore, s3transfer, boto3, sentencepiece, sacremoses, pytorch-transformers\n", + "Successfully installed boto3-1.17.97 botocore-1.20.97 jmespath-0.10.0 pytorch-transformers-1.2.0 s3transfer-0.4.2 sacremoses-0.0.45 sentencepiece-0.1.96\n", + "Collecting transformers\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/00/92/6153f4912b84ee1ab53ab45663d23e7cf3704161cb5ef18b0c07e207cef2/transformers-4.7.0-py3-none-any.whl (2.5MB)\n", + "\u001b[K |████████████████████████████████| 2.5MB 33.4MB/s \n", + "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n", + "Collecting tokenizers<0.11,>=0.10.1\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d4/e2/df3543e8ffdab68f5acc73f613de9c2b155ac47f162e725dcac87c521c11/tokenizers-0.10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3MB)\n", + "\u001b[K |████████████████████████████████| 3.3MB 42.8MB/s \n", + "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers) (20.9)\n", + "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers) (4.5.0)\n", + "Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers) (0.0.45)\n", + "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.19.5)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n", + "Collecting huggingface-hub==0.0.8\n", + " Downloading https://files.pythonhosted.org/packages/a1/88/7b1e45720ecf59c6c6737ff332f41c955963090a18e72acbcbeac6b25e86/huggingface_hub-0.0.8-py3-none-any.whl\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from transformers) (3.13)\n", + "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.0.12)\n", + "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.41.1)\n", + "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers) (2.4.7)\n", + "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.4.1)\n", + "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.7.4.3)\n", + "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n", + "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.0.1)\n", + "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2021.5.30)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n", + "Installing collected packages: tokenizers, huggingface-hub, transformers\n", + "Successfully installed huggingface-hub-0.0.8 tokenizers-0.10.3 transformers-4.7.0\n" + ] + } + ], + "source": [ + "import warnings\n", + "warnings.filterwarnings('ignore')\n", + "!pip install pytorch_transformers\n", + "!pip install transformers" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PRZoGwvSoQrb" + }, + "source": [ + "### Import the model to confirm if everything is fine." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "5YXfqVwh5arR" + }, + "outputs": [], + "source": [ + "from pytorch_transformers import BertModel\n", + "try :\n", + " import google.colab \n", + " model = BertModel.from_pretrained('finbert')\n", + " \n", + "except ModuleNotFoundError :\n", + " model = BertModel.from_pretrained('Data/finbert')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "r6XtCAXroYEh" + }, + "source": [ + "From here it will be he same as the IMDB_sentiment_classification and BioBERT_Demo notebooks respectively so we will not re-explain every step" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "5RgPXMbxblR-", + "outputId": "22c0c5e6-ae95-49ca-b70d-3806ce27ca64" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "6rYRtaycnfcO", - "colab_type": "text" - }, - "source": [ - "#Vocabulary\n", - "Finbert drive link does not have a vocab.txt file. In the FinBERT repo they have mentioned that the same vocab file was used as for the 'bert-base-uncased' model. So we are going to download the vocab.txt of the 'bert-base-uncased' model" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting pytorch_pretrained_bert\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d7/e0/c08d5553b89973d9a240605b9c12404bcf8227590de62bae27acbcfe076b/pytorch_pretrained_bert-0.6.2-py3-none-any.whl (123kB)\n", + "\r", + "\u001b[K |██▋ | 10kB 19.8MB/s eta 0:00:01\r", + "\u001b[K |█████▎ | 20kB 27.7MB/s eta 0:00:01\r", + "\u001b[K |████████ | 30kB 20.8MB/s eta 0:00:01\r", + "\u001b[K |██████████▋ | 40kB 23.0MB/s eta 0:00:01\r", + "\u001b[K |█████████████▎ | 51kB 24.3MB/s eta 0:00:01\r", + "\u001b[K |███████████████▉ | 61kB 20.8MB/s eta 0:00:01\r", + "\u001b[K |██████████████████▌ | 71kB 22.4MB/s eta 0:00:01\r", + "\u001b[K |█████████████████████▏ | 81kB 23.7MB/s eta 0:00:01\r", + "\u001b[K |███████████████████████▉ | 92kB 25.0MB/s eta 0:00:01\r", + "\u001b[K |██████████████████████████▌ | 102kB 26.4MB/s eta 0:00:01\r", + "\u001b[K |█████████████████████████████▏ | 112kB 26.4MB/s eta 0:00:01\r", + "\u001b[K |███████████████████████████████▊| 122kB 26.4MB/s eta 0:00:01\r", + "\u001b[K |████████████████████████████████| 133kB 26.4MB/s \n", + "\u001b[?25hCollecting pytorch-nlp\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/4f/51/f0ee1efb75f7cc2e3065c5da1363d6be2eec79691b2821594f3f2329528c/pytorch_nlp-0.5.0-py3-none-any.whl (90kB)\n", + "\r", + "\u001b[K |███▋ | 10kB 24.4MB/s eta 0:00:01\r", + "\u001b[K |███████▎ | 20kB 32.3MB/s eta 0:00:01\r", + "\u001b[K |███████████ | 30kB 36.3MB/s eta 0:00:01\r", + "\u001b[K |██████████████▌ | 40kB 36.8MB/s eta 0:00:01\r", + "\u001b[K |██████████████████▏ | 51kB 38.5MB/s eta 0:00:01\r", + "\u001b[K |█████████████████████▉ | 61kB 40.9MB/s eta 0:00:01\r", + "\u001b[K |█████████████████████████▌ | 71kB 41.5MB/s eta 0:00:01\r", + "\u001b[K |█████████████████████████████ | 81kB 43.0MB/s eta 0:00:01\r", + "\u001b[K |████████████████████████████████| 92kB 12.7MB/s \n", + "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (1.19.5)\n", + "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (4.41.1)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (2.23.0)\n", + "Requirement already satisfied: boto3 in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (1.17.97)\n", + "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (2019.12.20)\n", + "Requirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from pytorch_pretrained_bert) (1.8.1+cu101)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_pretrained_bert) (2.10)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_pretrained_bert) (3.0.4)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_pretrained_bert) (2021.5.30)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch_pretrained_bert) (1.24.3)\n", + "Requirement already satisfied: s3transfer<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from boto3->pytorch_pretrained_bert) (0.4.2)\n", + "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from boto3->pytorch_pretrained_bert) (0.10.0)\n", + "Requirement already satisfied: botocore<1.21.0,>=1.20.97 in /usr/local/lib/python3.7/dist-packages (from boto3->pytorch_pretrained_bert) (1.20.97)\n", + "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=0.4.1->pytorch_pretrained_bert) (3.7.4.3)\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.21.0,>=1.20.97->boto3->pytorch_pretrained_bert) (2.8.1)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.21.0,>=1.20.97->boto3->pytorch_pretrained_bert) (1.15.0)\n", + "Installing collected packages: pytorch-pretrained-bert, pytorch-nlp\n", + "Successfully installed pytorch-nlp-0.5.0 pytorch-pretrained-bert-0.6.2\n" + ] + } + ], + "source": [ + "#importing a few necessary packages and setting the DATA directory\n", + "DATA_DIR=\".\"\n", + "import os\n", + "import numpy as np\n", + "import pickle\n", + "import tensorflow as tf\n", + "import warnings\n", + "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n", + "\n", + "# install BERT\n", + "!pip install pytorch_pretrained_bert pytorch-nlp\n", + "\n", + "# BERT imports\n", + "import torch\n", + "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n", + "from keras.preprocessing.sequence import pad_sequences\n", + "from sklearn.model_selection import train_test_split\n", + "from pytorch_pretrained_bert import BertTokenizer, BertConfig\n", + "from pytorch_pretrained_bert import BertAdam, BertForSequenceClassification\n", + "from tqdm import tqdm, trange\n", + "import pandas as pd\n", + "import io\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "# specify GPU device\n", + "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", + "if torch.cuda.is_available() :\n", + " n_gpu = torch.cuda.device_count()\n", + " torch.cuda.get_device_name(0)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AWNqSJv4-26g" + }, + "source": [ + "You can download the dataset from [stochtwits](https://api.stocktwits.com/developers/docs). Please respect their policy of only 200 requests per hour.
\n", + "The dataset we created with the help of the API can be found under the Data folder in Ch10's folder of this repo " + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 73, + "resources": { + "http://localhost:8080/nbextensions/google.colab/files.js": { + "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK", + "headers": [ + [ + "content-type", + "application/javascript" + ] + ], + "ok": true, + "status": 200, + "status_text": "" + } + } }, + "id": "unBu59He5cSj", + "outputId": "d6649842-ce0a-4184-e257-1a665c6e67bc" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "STMiEGwTgg9F", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 217 - }, - "outputId": "2ce711f3-7081-44b9-8be9-5b0c527c6d2c" - }, - "source": [ - "!wget \"https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt\"" + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " Upload widget is only available when the cell has been executed in the\n", + " current browser session. Please rerun this cell to enable.\n", + " \n", + " " ], - "execution_count": 35, - "outputs": [ - { - "output_type": "stream", - "text": [ - "--2020-06-03 14:07:31-- https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt\n", - "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.217.46.38\n", - "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.217.46.38|:443... connected.\n", - "HTTP request sent, awaiting response... 200 OK\n", - "Length: 231508 (226K) [text/plain]\n", - "Saving to: ‘bert-base-uncased-vocab.txt’\n", - "\n", - "\r bert-base 0%[ ] 0 --.-KB/s \rbert-base-uncased-v 100%[===================>] 226.08K --.-KB/s in 0.05s \n", - "\n", - "2020-06-03 14:07:31 (4.33 MB/s) - ‘bert-base-uncased-vocab.txt’ saved [231508/231508]\n", - "\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "NmrwwAkToCz7", - "colab_type": "text" - }, - "source": [ - "### Move all the files into a FinBERT directory. This is the format that is required by the huggingface transformers library" + "text/plain": [ + "" ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" }, { - "cell_type": "code", - "metadata": { - "id": "hmjzyZEA48ZX", - "colab_type": "code", - "colab": {} - }, - "source": [ - "!mkdir finbert\n", - "!mv pytorch_model.bin finbert/\n", - "!mv config.json finbert/\n", - "!mv bert-base-uncased-vocab.txt vocab.txt\n", - "!mv vocab.txt finbert/" - ], - "execution_count": 0, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "MUCJlVHHoNd7", - "colab_type": "text" - }, - "source": [ - "### Installing packages" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Saving FinBERT_Data.csv to FinBERT_Data.csv\n" + ] + } + ], + "source": [ + "#importing the dataset\n", + "try :\n", + " # upload FinBERT_Data.csv\n", + " from google.colab import files\n", + " uploaded = files.upload()\n", + "\n", + "except ModuleNotFoundError :\n", + " print('Not using Colab') " + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "id": "n_KJMH-UOxw7" + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "try : \n", + " import google.colab \n", + " df = pd.read_csv(\"FinBERT_Data.csv\")\n", + "\n", + "except ModuleNotFoundError :\n", + " df = pd.read_csv(\"Data/FinBERT_Data.csv\")" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 195 }, + "id": "GClIpfVmPW_G", + "outputId": "cac4531c-2f1c-429d-dc36-fb6bc835b6ee" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "qsMk4azI5pcD", - "colab_type": "code", - "outputId": "243836c4-329c-4396-e0fb-8b80b880d71e", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - } - }, - "source": [ - "!pip install pytorch_transformers\n", - "!pip install transformers" + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
symbolsentimentmessagemessage_id
0GOOGLBullish$GOOGL want to know how to get more day trades...216640409
1GOOGLBullish$googl $amzn $fb\\nWhy we still bullish? Good v...216632254
2GOOGLBullish$tsla $aapl $googl\\nTsla ideas \\nhttps://youtu...216621697
3GOOGLBearish$AAPL $AMZN $GOOGL $FB $MSFT \\nTime to short t...216598529
4FBBearish2020 is not even close to being over.. \\nBig B...216673119
\n", + "
" ], - "execution_count": 6, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Collecting pytorch_transformers\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/a3/b7/d3d18008a67e0b968d1ab93ad444fc05699403fa662f634b2f2c318a508b/pytorch_transformers-1.2.0-py3-none-any.whl (176kB)\n", - "\u001b[K |████████████████████████████████| 184kB 2.8MB/s \n", - "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (1.18.4)\n", - "Collecting sacremoses\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n", - "\u001b[K |████████████████████████████████| 890kB 8.7MB/s \n", - "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (4.41.1)\n", - "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (1.13.19)\n", - "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (2.23.0)\n", - "Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (2019.12.20)\n", - "Collecting sentencepiece\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)\n", - "\u001b[K |████████████████████████████████| 1.1MB 15.9MB/s \n", - "\u001b[?25hRequirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from pytorch_transformers) (1.5.0+cu101)\n", - "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->pytorch_transformers) (1.12.0)\n", - "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->pytorch_transformers) (7.1.2)\n", - "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->pytorch_transformers) (0.15.1)\n", - "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_transformers) (0.3.3)\n", - "Requirement already satisfied: botocore<1.17.0,>=1.16.19 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_transformers) (1.16.19)\n", - "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_transformers) (0.10.0)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_transformers) (3.0.4)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_transformers) (2020.4.5.1)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_transformers) (2.9)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_transformers) (1.24.3)\n", - "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch>=1.0.0->pytorch_transformers) (0.16.0)\n", - "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.19->boto3->pytorch_transformers) (2.8.1)\n", - "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.19->boto3->pytorch_transformers) (0.15.2)\n", - "Building wheels for collected packages: sacremoses\n", - " Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893260 sha256=7871806c96a559cc4407566a5077b904b1026e57506fe2b6f2bfa76419e7a622\n", - " Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n", - "Successfully built sacremoses\n", - "Installing collected packages: sacremoses, sentencepiece, pytorch-transformers\n", - "Successfully installed pytorch-transformers-1.2.0 sacremoses-0.0.43 sentencepiece-0.1.91\n", - "Collecting transformers\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/48/35/ad2c5b1b8f99feaaf9d7cdadaeef261f098c6e1a6a2935d4d07662a6b780/transformers-2.11.0-py3-none-any.whl (674kB)\n", - "\u001b[K |████████████████████████████████| 675kB 2.9MB/s \n", - "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n", - "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.4)\n", - "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n", - "Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers) (0.0.43)\n", - "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.6/dist-packages (from transformers) (0.1.91)\n", - "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)\n", - "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)\n", - "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.18.4)\n", - "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.7)\n", - "Collecting tokenizers==0.7.0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/14/e5/a26eb4716523808bb0a799fcfdceb6ebf77a18169d9591b2f46a9adb87d9/tokenizers-0.7.0-cp36-cp36m-manylinux1_x86_64.whl (3.8MB)\n", - "\u001b[K |████████████████████████████████| 3.8MB 14.6MB/s \n", - "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (1.12.0)\n", - "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n", - "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n", - "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (0.15.1)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.9)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.4.5.1)\n", - "Installing collected packages: tokenizers, transformers\n", - "Successfully installed tokenizers-0.7.0 transformers-2.11.0\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "PRZoGwvSoQrb", - "colab_type": "text" - }, - "source": [ - "### Import the model to confirm if everything is fine." + "text/plain": [ + " symbol ... message_id\n", + "0 GOOGL ... 216640409\n", + "1 GOOGL ... 216632254\n", + "2 GOOGL ... 216621697\n", + "3 GOOGL ... 216598529\n", + "4 FB ... 216673119\n", + "\n", + "[5 rows x 4 columns]" ] + }, + "execution_count": 13, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 134 }, + "id": "0sHAcTXePZm8", + "outputId": "e4a72b57-22a6-41a4-fdf4-08b69f272b99" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "5YXfqVwh5arR", - "colab_type": "code", - "colab": {} - }, - "source": [ - "from pytorch_transformers import BertModel\n", - "model = BertModel.from_pretrained('finbert')" - ], - "execution_count": 0, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "r6XtCAXroYEh", - "colab_type": "text" - }, - "source": [ - "From here it will be he same as the IMDB_sentiment_classification and BioBERT_Demo notebooks respectively so we will not re-explain every step" + "data": { + "text/plain": [ + "FB 823\n", + "AMZN 671\n", + "GOOGL 620\n", + "Name: symbol, dtype: int64" ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" }, { - "cell_type": "code", - "metadata": { - "id": "5RgPXMbxblR-", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 508 - }, - "outputId": "d1406ab4-b6b6-480e-f1eb-47a74827f3b9" - }, - "source": [ - "#importing a few necessary packages and setting the DATA directory\n", - "DATA_DIR=\".\"\n", - "import os\n", - "import numpy as np\n", - "import pickle\n", - "import tensorflow as tf\n", - "import warnings\n", - "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n", - "\n", - "# install BERT\n", - "!pip install pytorch_pretrained_bert pytorch-nlp\n", - "\n", - "# BERT imports\n", - "import torch\n", - "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n", - "from keras.preprocessing.sequence import pad_sequences\n", - "from sklearn.model_selection import train_test_split\n", - "from pytorch_pretrained_bert import BertTokenizer, BertConfig\n", - "from pytorch_pretrained_bert import BertAdam, BertForSequenceClassification\n", - "from tqdm import tqdm, trange\n", - "import pandas as pd\n", - "import io\n", - "import numpy as np\n", - "import matplotlib.pyplot as plt\n", - "% matplotlib inline\n", - "\n", - "# specify GPU device\n", - "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", - "n_gpu = torch.cuda.device_count()\n", - "torch.cuda.get_device_name(0)" - ], - "execution_count": 8, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Collecting pytorch_pretrained_bert\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d7/e0/c08d5553b89973d9a240605b9c12404bcf8227590de62bae27acbcfe076b/pytorch_pretrained_bert-0.6.2-py3-none-any.whl (123kB)\n", - "\r\u001b[K |██▋ | 10kB 29.8MB/s eta 0:00:01\r\u001b[K |█████▎ | 20kB 1.7MB/s eta 0:00:01\r\u001b[K |████████ | 30kB 2.3MB/s eta 0:00:01\r\u001b[K |██████████▋ | 40kB 2.6MB/s eta 0:00:01\r\u001b[K |█████████████▎ | 51kB 2.0MB/s eta 0:00:01\r\u001b[K |███████████████▉ | 61kB 2.2MB/s eta 0:00:01\r\u001b[K |██████████████████▌ | 71kB 2.5MB/s eta 0:00:01\r\u001b[K |█████████████████████▏ | 81kB 2.7MB/s eta 0:00:01\r\u001b[K |███████████████████████▉ | 92kB 2.9MB/s eta 0:00:01\r\u001b[K |██████████████████████████▌ | 102kB 2.8MB/s eta 0:00:01\r\u001b[K |█████████████████████████████▏ | 112kB 2.8MB/s eta 0:00:01\r\u001b[K |███████████████████████████████▊| 122kB 2.8MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 133kB 2.8MB/s \n", - "\u001b[?25hCollecting pytorch-nlp\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/4f/51/f0ee1efb75f7cc2e3065c5da1363d6be2eec79691b2821594f3f2329528c/pytorch_nlp-0.5.0-py3-none-any.whl (90kB)\n", - "\r\u001b[K |███▋ | 10kB 28.7MB/s eta 0:00:01\r\u001b[K |███████▎ | 20kB 33.8MB/s eta 0:00:01\r\u001b[K |███████████ | 30kB 22.8MB/s eta 0:00:01\r\u001b[K |██████████████▌ | 40kB 17.8MB/s eta 0:00:01\r\u001b[K |██████████████████▏ | 51kB 12.1MB/s eta 0:00:01\r\u001b[K |█████████████████████▉ | 61kB 12.2MB/s eta 0:00:01\r\u001b[K |█████████████████████████▌ | 71kB 11.9MB/s eta 0:00:01\r\u001b[K |█████████████████████████████ | 81kB 11.7MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 92kB 6.7MB/s \n", - "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (1.18.4)\n", - "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (1.13.19)\n", - "Requirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (1.5.0+cu101)\n", - "Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (2019.12.20)\n", - "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (4.41.1)\n", - "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch_pretrained_bert) (2.23.0)\n", - "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_pretrained_bert) (0.10.0)\n", - "Requirement already satisfied: botocore<1.17.0,>=1.16.19 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_pretrained_bert) (1.16.19)\n", - "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch_pretrained_bert) (0.3.3)\n", - "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch>=0.4.1->pytorch_pretrained_bert) (0.16.0)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_pretrained_bert) (2.9)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_pretrained_bert) (2020.4.5.1)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_pretrained_bert) (3.0.4)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch_pretrained_bert) (1.24.3)\n", - "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.19->boto3->pytorch_pretrained_bert) (2.8.1)\n", - "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.19->boto3->pytorch_pretrained_bert) (0.15.2)\n", - "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.17.0,>=1.16.19->boto3->pytorch_pretrained_bert) (1.12.0)\n", - "Installing collected packages: pytorch-pretrained-bert, pytorch-nlp\n", - "Successfully installed pytorch-nlp-0.5.0 pytorch-pretrained-bert-0.6.2\n" - ], - "name": "stdout" - }, - { - "output_type": "stream", - "text": [ - "Using TensorFlow backend.\n" - ], - "name": "stderr" - }, - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "'Tesla P100-PCIE-16GB'" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 8 - } + "data": { + "text/plain": [ + "Bullish 1426\n", + "Bearish 688\n", + "Name: sentiment, dtype: int64" ] + }, + "execution_count": 14, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "display(df['symbol'].value_counts())\n", + "df['sentiment'].value_counts()" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "0MlChuiyPd5_", + "outputId": "22d94bf6-db06-443b-84ff-5800cadfbd8c" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "AWNqSJv4-26g", - "colab_type": "text" - }, - "source": [ - "You can download the dataset from [stochtwits](https://api.stocktwits.com/developers/docs). Please respect their policy of only 200 requests per hour.
\n", - "The dataset we created with the help of the API can be found under the Data folder in Ch10's folder of this repo " + "data": { + "text/plain": [ + "(2114, 4)" ] + }, + "execution_count": 15, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "df.shape" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 195 }, + "id": "LaJ50tg8QIFy", + "outputId": "75889d5a-0fa3-4ea2-fede-a98fadebef48" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "unBu59He5cSj", - "colab_type": "code", - "colab": { - "resources": { - "http://localhost:8080/nbextensions/google.colab/files.js": { - "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=", - "ok": true, - "headers": [ - [ - "content-type", - "application/javascript" - ] - ], - "status": 200, - "status_text": "" - } - }, - "base_uri": "https://localhost:8080/", - "height": 75 - }, - "outputId": "22080273-1d33-4eed-afb9-72bb27246f24" - }, - "source": [ - "#importing the dataset\n", - "from google.colab import files\n", - "uploaded = files.upload()" + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
symbolsentimentmessagemessage_id
0GOOGL1$GOOGL want to know how to get more day trades...216640409
1GOOGL1$googl $amzn $fb\\nWhy we still bullish? Good v...216632254
2GOOGL1$tsla $aapl $googl\\nTsla ideas \\nhttps://youtu...216621697
3GOOGL0$AAPL $AMZN $GOOGL $FB $MSFT \\nTime to short t...216598529
4FB02020 is not even close to being over.. \\nBig B...216673119
\n", + "
" ], - "execution_count": 9, - "outputs": [ - { - "output_type": "display_data", - "data": { - "text/html": [ - "\n", - " \n", - " \n", - " Upload widget is only available when the cell has been executed in the\n", - " current browser session. Please rerun this cell to enable.\n", - " \n", - " " - ], - "text/plain": [ - "" - ] - }, - "metadata": { - "tags": [] - } - }, - { - "output_type": "stream", - "text": [ - "Saving sentiment1_db.csv to sentiment1_db.csv\n" - ], - "name": "stdout" - } + "text/plain": [ + " symbol ... message_id\n", + "0 GOOGL ... 216640409\n", + "1 GOOGL ... 216632254\n", + "2 GOOGL ... 216621697\n", + "3 GOOGL ... 216598529\n", + "4 FB ... 216673119\n", + "\n", + "[5 rows x 4 columns]" ] + }, + "execution_count": 16, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "from sklearn.preprocessing import LabelEncoder\n", + "\n", + "#sentiment is positive and negative we need to convert it to 0,1\n", + "le = LabelEncoder()\n", + "df[\"sentiment\"] = le.fit_transform(df[\"sentiment\"])\n", + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "id": "C-L5hMUobhO5" + }, + "outputs": [], + "source": [ + "message = list(df['message'])\n", + "try :\n", + " import google.colab\n", + " tokenizer = BertTokenizer.from_pretrained('finbert', do_lower_case=True)\n", + "\n", + "except ModuleNotFoundError :\n", + " tokenizer = BertTokenizer.from_pretrained('Data/finbert', do_lower_case=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "cSp36svacg9b", + "outputId": "43aef6f2-fd48-4c63-bbf5-f6cf42471a44" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "n_KJMH-UOxw7", - "colab_type": "code", - "colab": {} - }, - "source": [ - "import pandas as pd\n", - "df = pd.read_csv(\"sentiment1_db.csv\")" - ], - "execution_count": 0, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "GClIpfVmPW_G", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 198 - }, - "outputId": "637c1eb7-1ac0-45a5-c808-7b44a78ed3c3" - }, - "source": [ - "df.head()" - ], - "execution_count": 16, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
symbolsentimentmessagemessage_id
0GOOGLBullish$GOOGL want to know how to get more day trades...216640409
1GOOGLBullish$googl $amzn $fb\\nWhy we still bullish? Good v...216632254
2GOOGLBullish$tsla $aapl $googl\\nTsla ideas \\nhttps://youtu...216621697
3GOOGLBearish$AAPL $AMZN $GOOGL $FB $MSFT \\nTime to short t...216598529
4FBBearish2020 is not even close to being over.. \\nBig B...216673119
\n", - "
" - ], - "text/plain": [ - " symbol ... message_id\n", - "0 GOOGL ... 216640409\n", - "1 GOOGL ... 216632254\n", - "2 GOOGL ... 216621697\n", - "3 GOOGL ... 216598529\n", - "4 FB ... 216673119\n", - "\n", - "[5 rows x 4 columns]" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 16 - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Tokenize the first sentence:\n", + "['[CLS]', '$', 'goo', '##gl', 'want', 'to', 'know', 'how', 'to', 'get', 'more', 'day', 'trades', 'with', 'google', 'and', 'any', 'other', 'stock', 'under', '25', '##k', 'and', '3', 'day', 'trades', 'in', '5', 'days', '?', 'try', 'this', 'youtube', '.', 'com', '/', 'watch', '?', 'v', '=', 'j', '##z', '##9', '_', 'c', '-', '5', '[SEP]']\n" + ] + } + ], + "source": [ + "# Restricting the max size of Tokens to 512(BERT doest accept any more than this)\n", + "tokenized_texts = list(map(lambda t: ['[CLS]']+tokenizer.tokenize(t)+['[SEP]'] , message))\n", + "print (\"Tokenize the first sentence:\")\n", + "print (tokenized_texts[0])\n" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "id": "nJ-2AXyhehxb" + }, + "outputs": [], + "source": [ + "classes = list(df['sentiment'])" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "id": "IIWG5cUDg-Je" + }, + "outputs": [], + "source": [ + "# Set the maximum sequence length. \n", + "MAX_LEN = 128\n", + "\n", + "# Pad our input tokens so that everything has a uniform length\n", + "input_ids = pad_sequences(list(map(tokenizer.convert_tokens_to_ids, tokenized_texts)),\n", + " maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "id": "XOVeJF_GhAW0" + }, + "outputs": [], + "source": [ + "# Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary\n", + "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]\n", + "input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "id": "KqtUrKn4hB7E" + }, + "outputs": [], + "source": [ + "# Create attention masks\n", + "attention_masks = []\n", + "# Create a mask of 1s for each token followed by 0s for padding\n", + "for seq in input_ids:\n", + " seq_mask = [float(i>0) for i in seq]\n", + " attention_masks.append(seq_mask)\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "id": "CKSuH5A8hD9T" + }, + "outputs": [], + "source": [ + "batch_size = 16\n", + "\n", + "# Use train_test_split to split our data into train and validation sets for training\n", + "train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(input_ids, classes, \n", + " random_state=2020, test_size=0.1)\n", + "train_masks, validation_masks, _, _ = train_test_split(attention_masks, input_ids,\n", + " random_state=2020, test_size=0.1)\n", + " \n", + "# Convert all of our data into torch tensors, the required datatype for our model\n", + "train_inputs = torch.tensor(train_inputs)\n", + "validation_inputs = torch.tensor(validation_inputs)\n", + "train_labels = torch.tensor(train_labels)\n", + "validation_labels = torch.tensor(validation_labels)\n", + "train_masks = torch.tensor(train_masks)\n", + "validation_masks = torch.tensor(validation_masks)\n", + "\n", + "# Create an iterator of our data with torch DataLoader \n", + "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n", + "train_sampler = RandomSampler(train_data)\n", + "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n", + "validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\n", + "validation_sampler = SequentialSampler(validation_data)\n", + "validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, + "id": "_yKqyfS3hGp6", + "outputId": "d6696421-b1e3-442e-f023-81633114c7d3" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "0sHAcTXePZm8", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 145 - }, - "outputId": "62fd7951-0a3a-4c12-8c75-65682b5629b5" - }, - "source": [ - "display(df['symbol'].value_counts())\n", - "df['sentiment'].value_counts()" - ], - "execution_count": 49, - "outputs": [ - { - "output_type": "display_data", - "data": { - "text/plain": [ - "FB 823\n", - "AMZN 671\n", - "GOOGL 620\n", - "Name: symbol, dtype: int64" - ] - }, - "metadata": { - "tags": [] - } - }, - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "1 1426\n", - "0 688\n", - "Name: sentiment, dtype: int64" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 49 - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "BertForSequenceClassification(\n", + " (bert): BertModel(\n", + " (embeddings): BertEmbeddings(\n", + " (word_embeddings): Embedding(30522, 768, padding_idx=0)\n", + " (position_embeddings): Embedding(512, 768)\n", + " (token_type_embeddings): Embedding(2, 768)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (encoder): BertEncoder(\n", + " (layer): ModuleList(\n", + " (0): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (1): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (2): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (3): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (4): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (5): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (6): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (7): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (8): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (9): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (10): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (11): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " )\n", + " )\n", + " (pooler): BertPooler(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (activation): Tanh()\n", + " )\n", + " )\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (classifier): Linear(in_features=768, out_features=2, bias=True)\n", + ")\n" + ] + } + ], + "source": [ + "#Loading pre trained BERT\n", + "try :\n", + " import google.colab\n", + " model = BertForSequenceClassification.from_pretrained(\"finbert\", num_labels=2)\n", + "\n", + "except ModuleNotFoundError :\n", + " model = BertForSequenceClassification.from_pretrained(\"Data/finbert\", num_labels=2)\n", + "\n", + "if torch.cuda.is_available() : \n", + " print(model.cuda())\n", + "\n", + "else :\n", + " print(model)" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 681 }, + "id": "e-YaYq_RhLK_", + "outputId": "b86fcf4c-b0d3-4757-bdea-18bdcd750001" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "0MlChuiyPd5_", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 35 - }, - "outputId": "53c30a9e-22ac-432d-dccf-eb42ac7897fe" - }, - "source": [ - "df.shape" - ], - "execution_count": 18, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "(2114, 4)" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 18 - } - ] + "name": "stderr", + "output_type": "stream", + "text": [ + "t_total value of -1 results in schedule not being applied\n", + "Epoch: 0%| | 0/4 [00:00\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
symbolsentimentmessagemessage_id
0GOOGL1$GOOGL want to know how to get more day trades...216640409
1GOOGL1$googl $amzn $fb\\nWhy we still bullish? Good v...216632254
2GOOGL1$tsla $aapl $googl\\nTsla ideas \\nhttps://youtu...216621697
3GOOGL0$AAPL $AMZN $GOOGL $FB $MSFT \\nTime to short t...216598529
4FB02020 is not even close to being over.. \\nBig B...216673119
\n", - "" - ], - "text/plain": [ - " symbol ... message_id\n", - "0 GOOGL ... 216640409\n", - "1 GOOGL ... 216632254\n", - "2 GOOGL ... 216621697\n", - "3 GOOGL ... 216598529\n", - "4 FB ... 216673119\n", - "\n", - "[5 rows x 4 columns]" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 19 - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Train loss: 0.6394780593759873\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "C-L5hMUobhO5", - "colab_type": "code", - "colab": {} - }, - "source": [ - "message = list(df['message'])\n", - "tokenizer = BertTokenizer.from_pretrained('finbert', do_lower_case=True)" - ], - "execution_count": 0, - "outputs": [] + "name": "stderr", + "output_type": "stream", + "text": [ + "\r", + "Epoch: 25%|██▌ | 1/4 [00:59<02:57, 59.17s/it]" + ] }, { - "cell_type": "code", - "metadata": { - "id": "cSp36svacg9b", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 74 - }, - "outputId": "382434f8-b44a-4fab-d42d-f55ae7b18c3c" - }, - "source": [ - "# Restricting the max size of Tokens to 512(BERT doest accept any more than this)\n", - "tokenized_texts = list(map(lambda t: ['[CLS]']+tokenizer.tokenize(t)+['[SEP]'] , message))\n", - "print (\"Tokenize the first sentence:\")\n", - "print (tokenized_texts[0])\n" - ], - "execution_count": 38, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Tokenize the first sentence:\n", - "['[CLS]', '$', 'goo', '##gl', 'want', 'to', 'know', 'how', 'to', 'get', 'more', 'day', 'trades', 'with', 'google', 'and', 'any', 'other', 'stock', 'under', '25', '##k', 'and', '3', 'day', 'trades', 'in', '5', 'days', '?', 'try', 'this', 'youtube', '.', 'com', '/', 'watch', '?', 'v', '=', 'j', '##z', '##9', '_', 'c', '-', '5', '[SEP]']\n" - ], - "name": "stdout" - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation Accuracy: 0.7232142857142857\n", + "Train loss: 0.4459469406043782\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "nJ-2AXyhehxb", - "colab_type": "code", - "colab": {} - }, - "source": [ - "classes = list(df['sentiment'])" - ], - "execution_count": 0, - "outputs": [] + "name": "stderr", + "output_type": "stream", + "text": [ + "\r", + "Epoch: 50%|█████ | 2/4 [01:58<01:58, 59.13s/it]" + ] }, { - "cell_type": "code", - "metadata": { - "id": "IIWG5cUDg-Je", - "colab_type": "code", - "colab": {} - }, - "source": [ - "# Set the maximum sequence length. \n", - "MAX_LEN = 128\n", - "\n", - "# Pad our input tokens so that everything has a uniform length\n", - "input_ids = pad_sequences(list(map(tokenizer.convert_tokens_to_ids, tokenized_texts)),\n", - " maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")\n", - "\n" - ], - "execution_count": 0, - "outputs": [] + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation Accuracy: 0.75\n", + "Train loss: 0.16766617303135015\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "XOVeJF_GhAW0", - "colab_type": "code", - "colab": {} - }, - "source": [ - "# Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary\n", - "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]\n", - "input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")" - ], - "execution_count": 0, - "outputs": [] + "name": "stderr", + "output_type": "stream", + "text": [ + "\r", + "Epoch: 75%|███████▌ | 3/4 [02:57<00:59, 59.18s/it]" + ] }, { - "cell_type": "code", - "metadata": { - "id": "KqtUrKn4hB7E", - "colab_type": "code", - "colab": {} - }, - "source": [ - "# Create attention masks\n", - "attention_masks = []\n", - "# Create a mask of 1s for each token followed by 0s for padding\n", - "for seq in input_ids:\n", - " seq_mask = [float(i>0) for i in seq]\n", - " attention_masks.append(seq_mask)\n", - " " - ], - "execution_count": 0, - "outputs": [] + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation Accuracy: 0.6919642857142857\n", + "Train loss: 0.06754684608777937\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "CKSuH5A8hD9T", - "colab_type": "code", - "colab": {} - }, - "source": [ - "batch_size = 16\n", - "\n", - "# Use train_test_split to split our data into train and validation sets for training\n", - "train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(input_ids, classes, \n", - " random_state=2020, test_size=0.1)\n", - "train_masks, validation_masks, _, _ = train_test_split(attention_masks, input_ids,\n", - " random_state=2020, test_size=0.1)\n", - " \n", - "# Convert all of our data into torch tensors, the required datatype for our model\n", - "train_inputs = torch.tensor(train_inputs)\n", - "validation_inputs = torch.tensor(validation_inputs)\n", - "train_labels = torch.tensor(train_labels)\n", - "validation_labels = torch.tensor(validation_labels)\n", - "train_masks = torch.tensor(train_masks)\n", - "validation_masks = torch.tensor(validation_masks)\n", - "\n", - "# Create an iterator of our data with torch DataLoader \n", - "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n", - "train_sampler = RandomSampler(train_data)\n", - "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n", - "validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\n", - "validation_sampler = SequentialSampler(validation_data)\n", - "validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)" - ], - "execution_count": 0, - "outputs": [] + "name": "stderr", + "output_type": "stream", + "text": [ + "Epoch: 100%|██████████| 4/4 [03:56<00:00, 59.16s/it]" + ] }, { - "cell_type": "code", - "metadata": { - "id": "_yKqyfS3hGp6", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "outputId": "91099b6b-f9dd-45fa-f76d-4dbcbfbaebfd" - }, - "source": [ - "#Loading pre trained BERT\n", - "model = BertForSequenceClassification.from_pretrained(\"finbert\", num_labels=2)\n", - "model.cuda()" - ], - "execution_count": 46, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "BertForSequenceClassification(\n", - " (bert): BertModel(\n", - " (embeddings): BertEmbeddings(\n", - " (word_embeddings): Embedding(30522, 768, padding_idx=0)\n", - " (position_embeddings): Embedding(512, 768)\n", - " (token_type_embeddings): Embedding(2, 768)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (encoder): BertEncoder(\n", - " (layer): ModuleList(\n", - " (0): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (1): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (2): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (3): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (4): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (5): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (6): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (7): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (8): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (9): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (10): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (11): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " )\n", - " )\n", - " (pooler): BertPooler(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (activation): Tanh()\n", - " )\n", - " )\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " (classifier): Linear(in_features=768, out_features=2, bias=True)\n", - ")" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 46 - } - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation Accuracy: 0.7321428571428571\n" + ] }, { - "cell_type": "code", - "metadata": { - "id": "e-YaYq_RhLK_", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 876 - }, - "outputId": "7e4fb24e-3c35-421c-be11-c3e7ab1f0f37" - }, - "source": [ - "# BERT fine-tuning parameters\n", - "param_optimizer = list(model.named_parameters())\n", - "no_decay = ['bias', 'gamma', 'beta']\n", - "optimizer_grouped_parameters = [\n", - " {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n", - " 'weight_decay_rate': 0.01},\n", - " {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n", - " 'weight_decay_rate': 0.0}\n", - "]\n", - "\n", - "optimizer = BertAdam(optimizer_grouped_parameters,\n", - " lr=2e-5,\n", - " warmup=.1)\n", - "\n", - "# Function to calculate the accuracy of our predictions vs labels\n", - "def flat_accuracy(preds, labels):\n", - " pred_flat = np.argmax(preds, axis=1).flatten()\n", - " labels_flat = labels.flatten()\n", - " return np.sum(pred_flat == labels_flat) / len(labels_flat)\n", - "\n", - "torch.cuda.empty_cache() \n", - "# Store our loss and accuracy for plotting\n", - "train_loss_set = []\n", - "# Number of training epochs \n", - "epochs = 4\n", - "\n", - "# BERT training loop\n", - "for _ in trange(epochs, desc=\"Epoch\"): \n", - " \n", - " ## TRAINING\n", - " \n", - " # Set our model to training mode\n", - " model.train() \n", - " # Tracking variables\n", - " tr_loss = 0\n", - " nb_tr_examples, nb_tr_steps = 0, 0\n", - " # Train the data for one epoch\n", - " for step, batch in enumerate(train_dataloader):\n", - " # Add batch to GPU\n", - " batch = tuple(t.to(device) for t in batch)\n", - " # Unpack the inputs from our dataloader\n", - " b_input_ids, b_input_mask, b_labels = batch\n", - " # Clear out the gradients (by default they accumulate)\n", - " optimizer.zero_grad()\n", - " # Forward pass\n", - " loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n", - " train_loss_set.append(loss.item()) \n", - " # Backward pass\n", - " loss.backward()\n", - " # Update parameters and take a step using the computed gradient\n", - " optimizer.step()\n", - " # Update tracking variables\n", - " tr_loss += loss.item()\n", - " nb_tr_examples += b_input_ids.size(0)\n", - " nb_tr_steps += 1\n", - " print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n", - " \n", - " ## VALIDATION\n", - "\n", - " # Put model in evaluation mode\n", - " model.eval()\n", - " # Tracking variables \n", - " eval_loss, eval_accuracy = 0, 0\n", - " nb_eval_steps, nb_eval_examples = 0, 0\n", - " # Evaluate data for one epoch\n", - " for batch in validation_dataloader:\n", - " # Add batch to GPU\n", - " batch = tuple(t.to(device) for t in batch)\n", - " # Unpack the inputs from our dataloader\n", - " b_input_ids, b_input_mask, b_labels = batch\n", - " # Telling the model not to compute or store gradients, saving memory and speeding up validation\n", - " with torch.no_grad():\n", - " # Forward pass, calculate logit predictions\n", - " logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask) \n", - " # Move logits and labels to CPU\n", - " logits = logits.detach().cpu().numpy()\n", - " label_ids = b_labels.to('cpu').numpy()\n", - " tmp_eval_accuracy = flat_accuracy(logits, label_ids) \n", - " eval_accuracy += tmp_eval_accuracy\n", - " nb_eval_steps += 1\n", - " print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n", - "\n", - "# plot training performance\n", - "plt.figure(figsize=(15,8))\n", - "plt.title(\"Training loss\")\n", - "plt.xlabel(\"Batch\")\n", - "plt.ylabel(\"Loss\")\n", - "plt.plot(train_loss_set)\n", - "plt.show()" - ], - "execution_count": 50, - "outputs": [ - { - "output_type": "stream", - "text": [ - "t_total value of -1 results in schedule not being applied\n", - "\n", - "\n", - "Epoch: 0%| | 0/4 [00:00" - ] - }, - "metadata": { - "tags": [], - "needs_background": "light" - } - } - ] + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] }, { - "cell_type": "markdown", - "metadata": { - "id": "78yvfEVHoncI", - "colab_type": "text" - }, - "source": [ - "We achieved a validation accuracy of 73% in the last epoch. This is just a demonstration, we might be able to achieve higher accuracies by performing some pre-processing steps and hyper parameter tuning." + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3gAAAHwCAYAAAD0Es3SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy9ebQk113n+b2xZL736tUiqbRZi3eMzQ4ya4/bDdNgm8XM0GwH6GbOgKFZxjMw0Ex3D3CY7gMHc9iFjYHpbsxi6AGMAdsyGNmSbdlYlmRbq7VLVSqpqlT16q2ZGRH3zh8R98aNyIjIyMyIl9v3c47Oq5cZLyIyMt7T/eb39/v+hFIKhBBCCCGEEEIWH2fWJ0AIIYQQQgghpBko8AghhBBCCCFkSaDAI4QQQgghhJAlgQKPEEIIIYQQQpYECjxCCCGEEEIIWRIo8AghhBBCCCFkSaDAI4QQshIIId4rhPg3TW875jm8Vghxqun9EkIIIRpv1idACCGElCGE2LW+3QDQBxAl3/+QUuqP6+5LKfX6NrYlhBBC5gkKPEIIIXOLUmpT/1sI8QSAH1BK/UN+OyGEp5QKD/PcCCGEkHmEJZqEEEIWDl3qKIT4d0KIZwH8FyHEZUKIvxVCnBNCXEz+fb31Mx8UQvxA8u/vF0J8WAjxK8m2jwshXj/hti8WQtwmhNgRQvyDEOJmIcQf1Xwdr0yOtSWEuE8I8S3Wc28QQtyf7Pe0EOL/TB4/mby2LSHEBSHE7UII/v+cEEIIAAo8Qgghi8s1AC4H8EIAb0L8/7T/knx/I4ADAL9d8fNfAeAhACcB/DKAPxBCiAm2/RMA/wTgCgA/D+D76py8EMIH8DcA3g/gKgA/DuCPhRCvSDb5A8RlqEcBfD6Af0we/0kApwBcCeBqAP8egKpzTEIIIcsPBR4hhJBFRQL4OaVUXyl1oJR6Xin1F0qpfaXUDoD/DOCfV/z8k0qp31NKRQD+G4BrEQum2tsKIW4E8GoAP6uUGiilPgzg3TXP/ysBbAL4peRn/xHA3wL47uT5AMCrhBDHlFIXlVJ3WY9fC+CFSqlAKXW7UooCjxBCCAAKPEIIIYvLOaVUT38jhNgQQvyuEOJJIcQ2gNsAnBBCuCU//6z+h1JqP/nn5pjbvgDABesxAHi65vm/AMDTSilpPfYkgOuSf38bgDcAeFII8SEhxFclj78FwCMA3i+EeEwI8TM1j0cIIWQFoMAjhBCyqORdq58E8AoAX6GUOgbgNcnjZWWXTXAGwOVCiA3rsRtq/uwzAG7I9c/dCOA0ACilPqGUeiPi8s13Afjz5PEdpdRPKqVeAuBbAPyEEOLrpnwdhBBClgQKPEIIIcvCUcR9d1tCiMsB/FzbB1RKPQngTgA/L4ToJC7bN9f88Y8D2Afw00IIXwjx2uRn35ns63uEEMeVUgGAbcQlqRBCfJMQ4mVJD+AlxGMjZPEhCCGErBoUeIQQQpaFXwewDuA8gI8BeN8hHfd7AHwVgOcB/CcAf4Z4Xl8lSqkBYkH3esTn/DsA/rVS6sFkk+8D8ERSbvrDyXEA4OUA/gHALoA7APyOUurWxl4NIYSQhUawL5sQQghpDiHEnwF4UCnVuoNICCGE5KGDRwghhEyBEOLVQoiXCiEcIcTrALwRcc8cIYQQcuh4sz4BQgghZMG5BsBfIp6DdwrAv1VK3T3bUyKEELKqsESTEEIIIYQQQpYElmgSQgghhBBCyJJAgUcIIYQQQgghS8LC9eCdPHlSvehFL5r1aRBCCCGEEELITPjkJz95Xil1ZdFzCyfwXvSiF+HOO++c9WkQQgghhBBCyEwQQjxZ9hxLNAkhhBBCCCFkSaDAI4QQQgghhJAlgQKPEEIIIYQQQpYECjxCCCGEEEIIWRIo8AghhBBCCCFkSaDAI4QQQgghhJAlgQKPEEIIIYQQQpYECjxCCCGEEEIIWRIo8AghhBBCCCFkSaDAI4QQQgghhJAlgQKPEEIIIYQQQpYECjxCCCGEEEIIWRIo8AghhBBCCCFkSaDAI4QQQgghhJAlgQKPEEIIIYQQQpYECjxCCCGEEEIIWRIo8AghhBBCCCFkSaDAI3PPn33iKXzDr90269MghBBCCCFk7qHAI3PP4+f38dBzO5BSzfpUCCGEEEIImWso8Mjco1Qs7AaRnPGZEEIIIYQQMt9Q4JG5RyYCrx9Q4BFCCCGEEFIFBR6Ze3RlZi+MZnsihBBCCCGEzDkUeGTuoYNHCCGEEEJIPSjwyNyT6Dv06eARQgghhBBSCQUemXu0g9ejg0cIIYQQQkglFHhk7jElmnTwCCGEEEIIqaQ1gSeE+H+FEGeFEPeWPC+EEL8phHhECPFpIcSXtnUuZLGRpkSTDh4hhBBCCCFVtOng/VcAr6t4/vUAXp789yYAb23xXMgCo0yJJh08QgghhBBCqmhN4CmlbgNwoWKTNwL4QxXzMQAnhBDXtnU+ZHGRiXFHB48QQgghhJBqZtmDdx2Ap63vTyWPkRqc2+nj4t5g1qdxKLAHjxBCCCGEkHosRMiKEOJNQog7hRB3njt3btanMxe8+Z1342fffd+sT+NQiJiiSQghhBBCSC1mKfBOA7jB+v765LEhlFJvV0rdpJS66corrzyUk5t3tvYDnNvpzfo0DgUzB489eIQQQgghhFQyS4H3bgD/OknT/EoAl5RSZ2Z4PguFVAoHg9UQPGmJJh08QgghhBBCqvDa2rEQ4k8BvBbASSHEKQA/B8AHAKXU2wC8B8AbADwCYB/A/9LWuSwjUinsr4zAi7+yRJMQQgghhJBqWhN4SqnvHvG8AvCjbR1/2YmkWhnBw5AVQgghhBBC6rEQISuLxBPn9/BL733QzG5rC6WAgxXpSVMs0SSEEEIIIaQWFHgN84EHz+JtH3oUF/eDVo8TKYX9QdjqMeYFPQePg84JIYQQQgiphgKvYYIoViOyZQdPqrhEU8p2jzMPMGSFEEIIIYSQelDgNUyoBV7Lwsu4WivQl6YvJQUeIYQQQggh1VDgNUwQxWqkbWNNu1qrkKSpzKDz5X+thBBCCCGETAMFXsOEibUWtVyiGSUKchVm4bFEkxBCCCGEkHpQ4DWMcfDaLtFMdr8KDp4p0aSDR8jYKKVw7+lLsz4NQgghhBwSFHgNc1ghK8qUaC5/kqa+lj06eISMzV1PXcQ3/daH8dCzO7M+FUIIIYQcAhR4DRMeUg+eLgFdhRJNRQePkInZPog/BNrptTu6hRBCCCHzAQVew2gHL2o9RXN1Qla0gzegg0fI2OjfnxWYqEIIIYQQUOA1ju7BU63PwYu/HqyAqyWZoknIxOgPm9ouGyeEEELIfECB1zCHlaIpV6hEk3PwCJkc/ftDgUcIIYSsBhR4DWNCVlrWIpFcnZAVxTEJhEyM/v2hviOEEEJWAwq8hkkHnbedohl/3V+BskXtQLBEk5DxoYNHCCGErBYUeA0THtKYhFVK0dTXMpTKXF9CSD0iOniEEELISkGB1zBh8nF56ymaapVSNNN/DyjwCBkLpRiyQgghhKwSFHgNo6P829R3SinzafwqpGjaiaS9gAKPkHGQdPAIIYSQlYICr2HCQ4gkt8XjKpVoAkA/XP7XS0iTaNObDh4hhBCyGlDgNYzpwWvRwrPLP1chRdNOJO3TwSNkLDjonBBCCFktKPAaZpCkaLY5B8/+JH41evCsEk06eISMBXvwCCGEkNWCAq9htIPX5lrKXqitQommUkDHi29VOniEjId27hQFHiGEELISUOA1zGGkaNq7XhUHb913AXDYOSHjEkmWaBJCCCGrBAVewwSHMAcv4+CtQIqmLfA47JyQ8WCJJiGEELJaUOA1zKEIPLl6JZobHTp4hEyC/nNBB48QQghZDSjwGiZMQlZkizpEL9QcsSIpmkphzZRoLr+gJaRJdIkme/AIIYSQ1YACr2G0g9dmiqZesB3peitSogmsd3SJJh08QsZBskSTEEIIWSko8BomiNr/tFzve7PrIYiUEZXLSjZkZfkFLSFNov8UtVlVQAghhJD5gQKvYcJkFdWm5oosgQcsf5Kmshw8jkkgZDzo4BFCCCGrBQVegyiljIPXbopm/PVIIvCWPWglk6JJB4+QsdAfCFHfEUIIIasBBV6D2LPvDiNF8+iadvCWO2glDlnhoHNCJsGUaFLhEUIIISsBBV6DaPcOOJw5eEc6iYO35EErkQRcx0HHczgmgZAxkRx0TgghhKwUFHgNElgpBq324CUrtc211SjRVErBEUDXczjonJAxidiDRwghhKwUFHgNEh6agxd/XZWQFakUHCGw2fWw21/uclRCmkb/veAcPEIIIWQ1oMBrEHtcgWyxHkqLxw0zG27ZBV481P34uo9LB8GsT4eQhUIplmgSQgghqwQFXoNkBF6Liykt8Dpe/PaFS75yk0pBCIFjaz62KfAIGQuOSSCEEEJWCwq8BrFLNKMWF1O6B29VBJ5SgCMEjtHBI2Rs9OdOS/5nghBCCCEJFHgNElohK232u+hdd9xE4LWZ6DIHSKXgOnGJJh08QsZDmTl4VHiEEELIKkCB1yCD0HLwDqEHr6sdvGi5F246ZIU9eISMD0s0CSGEkNWCAq9BbAevzXKoVSvRlAoQicDbG0SZXkdCSDUs0SSEEEJWCwq8BskMOm/VwYu/pgJvuQWPnoN3bD0eC7HT46gEQupCB48QQghZLSjwGiSbotl+iWbHdZPjLvfCTSYhK8fXfQBgmSYhY5D24M34RAghhBByKFDgNUjbKZr3PXMJO73AuIPawYuW3MGTiYNHgUfI+GjHv82qAkIIIYTMDxR4DRJkUjSb3bdSCt/21o/ijz/+lBGPOmRlmR08pRSU1YMHUOARMg4RB50TQgghKwUFXoNkHLyGV1NSAb1AYrcXpmMSjIO3vCs3/VrtEk2OSiCkPoo9eIQQQshKQYHXIG324GkRFyll/u05AkIs9xw8fR3jkJXVc/DufOICdvsMlSGTowsLOAePEEIIWQ0o8BokI/Aad/ASgSdVKnocAd9xECyxg6dfmuOsXonm/iDEd779Y/jLu07N+lTIAiNZokkIIYSsFBR4DWKXaDa9mNIfvmcEnhBwHbHUJZr6tQoBrPkuOp6zMiWag1Aikgp7/Wim5/GJJy7gW377w+iHsz0PMhkRSzQJIYSQlYICr0FaLdG0HbzkMK4Q8Fyx1IO/7R48IE7SXBUHTw+wn/XC/P5ntvHpU5dwaX81rvuyoW+fJf4ciBBCCCEWFHgNYpdKNj0mwS7RjCxXy3edjHO4bNg9eEAs8LZ7qyE0TN/ljFfm5t6jA7SQ6PePPXiEEELIakCB1yB22EnjYxKSXUdKmYWa68QlmuESfzRvl6MCwLE1b+UcvFm/v5FxEmd6GmRCojlxggkhhBByOFDgNUi7YxISFyVS0DrSEQK+I6ZO0fzDO57At7/to1OeYTvoyyhWsEQzSu6nWQ+oNiV+VHgLCUs0CSGEkNWCAq9BBonQ6npOez14Kg1ZcR3Ac52pHZ7PPreDu57amssFvCoo0VwVgRckzZazLo20+z/J4pGWaM74RAghhBByKFDgNYh28DqecyhjEoQQ8Boo0YxkvN+d3vzNWzNjEmwHb0XCPkxpHXvwyBSkYxL4/hFCCCGrAAVeg4RSwkmCTw5jTIJO0Zy2RDNKnKIL+4Op9tMGRSErO/1w5qLnMNAfGMy6B0/OidAkkxFx0DkhhBCyUlDgNcggkvBcB44QjbsddqKitHrwXGf6Ek398xf25lfg6R68Y+s+lAJ2+vPnNjbN/KRoZr+SxUIZB2/GJ0IIIYSQQ4ECr0HCSMF3BBzR/KflRWMSHAfwG3Hw4v1tzaGDl5+Dd2zdB4CVGHYeJkp+1qV18yI0yWSwRJMQQghZLSjwGiSMJHzPgeuIxhfDem0WSmUFjzTTg7cIDp4u0bz+snUAwD1Pb83qlA4NfQ/NukRTUSAsNBEdWEIIIWSloMBrkEAqeE5cotn0YsqeZaUNO9cR8JzpB53rOP6Lc+jg5UNWvvLFV+DGyzfwjjuenOFZHQ7hnPS+MUVzsVEmRZPvHyGEELIKUOA1SBBK+K6A49RblEdS4Q8+/Dh6QTRyW+2ehJkUTcQhK3LKEk2lHbz5K3vU1zHRd3Acge/7yhfin564gAfObM/wzNpnXkoj9eGZormYsESTEEIIWS0o8BoklAq+qx280Yupz5y+hP/nb+/HHY89P3JbE3RhCTxHCHiug2BaB2+BevAA4Ntvuh5dz8E7PrbcLl44LwJP0gFaZPTnPzRgCSGEkNXAm/UJLBNBJOG5AlBp30sV/cS5C8LRDlzq4Emz4HaTHrxpBcBC9OBZH0Wc2Ojgn73sJO568uKMzupwiOZk0Hka8DPT0yATEtHBI4QQQlYKOngNEkQSvuNAiHqLKe281RFopsxKpuJRh6wEDc3Bm88evNSttLnq2BrO7/ZncUqHRjjG/dEm+vaa9XmQyUh78GZ8IoQQQgg5FCjwGiSMFDxXwHVErR68IBFWdVISdZlVpKwUTUf34E3p4EXz7ODFX0VO4F252cGFvcFSiw47WGeWsIdrsUnnGDbz/v31PaeX/sMVQgghZJGhwGuQYMwevHEcGjtkRW8fO3jO1CIn7cGbv5CVdCRE9vErj3Yh1XyK0qbQwn3alNRpsWcwksUj/aBg+n3t9kO8+Z334F13n55+Z4QQQghpBQq8BgmjOEVTCFGrX0mXVtZy8EyJpjILNdcR8NzpSzT18S/uD2YeyZ8nPyZBc3KzCwA4t7O8TgIdPNIETc4xDJO/NdMGOxFCCCGkPSjwGiSIJDzHgevUSxzUwiyqMeZAC52hMQkNhKykC3hguzdfLl5+0Lnm5NFY4C1zqdi8pGhGJoWRi/pFRN8+TaSgzsuHDoQQQggphwKvQYJIwffiEs06yYf6U/A6Dp5ZWEmVTdFsYEyCXQI4byWP+nXne/C0g7fMAi8ao0ezTRRTNBcaO6BpWiKrkoAQQggh8wkFXoOEUsJ3RNKDV2N74+CN3liZHjxpFlk6RXPqQedSYd13AQAX56wPr2gOHgCc3OwAWG6BF86JWzIvA9fJZDTputlhT4QQQgiZT1oVeEKI1wkhHhJCPCKE+JmC528UQtwqhLhbCPFpIcQb2jyftgnCOEXTEfU+4TY9eDUcuDQJz+pLc5KQlWkdPClx8mgsmC7OmYOnF6Vu7k7d7Hroeg7O787X+TbJvIxJaDqFsW0eObtLh8lCWX87poUOHiGEEDL/tCbwhBAugJsBvB7AqwB8txDiVbnN/iOAP1dKfQmA7wLwO22dz2EQSAnPdeIxCS3NwQulhFLK9KT5rjDjFiYlkgpXJiWPF+ZsFl7ab5h18IQQOLnZxfklDlmZlx68RQpZOXPpAP/y1z6EDz18btanMjfo962RHjz9N2sB7gVCCCFkVWnTwftyAI8opR5TSg0AvBPAG3PbKADHkn8fB/BMi+fTOmGk0HGdJEVzDAev1hw8a9C5VHAThec6YuoY/VAqXJmElsyfgxd/zZdoAnHQyrklLtHUPXjzIvBmfR512D4IoRRwac5KjWdJ1KBAj9iPSQghhMw9bQq86wA8bX1/KnnM5ucBfK8Q4hSA9wD48RbPp3XiFE0BVwjUWUulDs04KZoSUqWOluc6CKWa6tN5KRWOrfk42vVw60Nn52ohXzYHD4iHnS91iaa+P2b8dixSciL7BYdptERTNucGEkIIIaQdZh2y8t0A/qtS6noAbwDwDiHE0DkJId4khLhTCHHnuXPzW3oVRAqe68Bx6pUwDcLx5+BFMv63mwg8P1E+0yxoQxn3Dv7Hb3olPvbYBfza33924n01TaWDt9ld7jl40Xz0O+lbeRFcG/N7QgFiaLLEdpHcXEIIIWRVaVPgnQZwg/X99cljNv8rgD8HAKXUHQDWAJzM70gp9Xal1E1KqZuuvPLKlk53ct76wUfx5nfejVBKdFydojl6ARRWlOA9cGYbj5/fM9+npVESUqY9eK4rkn1NvuDSJZ/f+eob8W1fej1++9ZH5kY42TP/8pzc7OLCXn9pF5v6PZ31mAR7RMe8s0jnelikPXjT78s4pBTQhBBCyNzSpsD7BICXCyFeLIToIA5ReXdum6cAfB0ACCFeiVjgza9FV8LBIMS7P/UMdnth7OAJUTNFs3wB/9P/36fxS+99wHyvrE/OI6XgONrBc0r3UZdQKnjJfl7zObG+vnQwHz1M6aDzIgevA6mAi1MGw7zr7tN4w2/cPndlZ02JlUEo8ei53Yl/fpFCVujgDSMbHFRPAU0IIYTMP60JPKVUCODHANwC4AHEaZn3CSF+QQjxLclmPwngB4UQnwLwpwC+X83bKrsGb/yS66BUWuoYp2iO/rmgYg7ehb0Btg9C872ZPyUVlEoFjw5bCaeon7NDWzrJPAJdPjpryubgAXHICjD9LLwPPHgW95/ZnjsnMGzILXnXPafx+l+/Hbv9cPTGBSySaEqHes//uR4WrZRoLsC9QAghhKwqXps7V0q9B3F4iv3Yz1r/vh/A17R5DofBS6/cxBddfxyfOnUJHdeBI+r1qFTNwdvpBTgIIvN9ZC2sbEHmN1CiGco4HAYAOp6TObdZkzp4w8+dTEY7nN8ZANdMfowHzmwD0AJ98v00TVMpmpf2Awwiif1BiM3u+L/yZg7eAoimyPogZJG59cGz+KOPPYk/+P5XT72vVOBNvSsrGGr6fRFCCCGkHWYdsrI0vPGL44BQz3Hq9+CZOXjZ1ZJSCrv9ED1L4CnjTMQLNi14vMRxm2ZUgpSpE6gF3mBOVnB6UZqfgwdYAm8KB68XRHgsKV+cN1HQ1Bw8vZ9gwntkkZIp0x6xGZ/IlNz11EV84MGzjZQN67etkX0xRZMQQgiZeyjwGuKbv+gF6HoOLj/i1xZ4g5I5ePuDCFIh4+BlxySooRLNaRy3UErLEZyvEs0qB0+7UXuDyUoPAeChZ3fSaztnqqApYaXLd4MJ39O0LG+q0zgUlqVEMx1NMf2+ZIP7WiSxTwghhKwqFHgNceXRLj74U6/Fd776xto9eKmDl914pxcLloOBVaJpLdIimQo8XaI56YJLSgWp5tfBS+fgDSs8U046hRjV5ZlAmmo6S3b7IX72r+/FXj9s3MGb9D1NUxjnf1G/LCmPTQ4nb7IHL2IPHiGEEDL3UOA1yLXH19HxHAhRz0HQgiLv4O324wTLrIOXbhNGaQ+eZ1I0J1u864WaN6chK/plVQm8acSoLfDmwZW456kt/OEdT+JTT2+ZOXjTLqb165r0PZUL1NeWjhOZ/3OtInXdmivRbMYN1F8X+/oSQgghywwFXgvEDl6NEs2weDG6nTh42R486+ciaebCeaZEc7r+KjcRivMaslI0B68JMfrAmR3z72AOFq36ug8iaYT/tIvptAdvug8BFsG1UUtTohl/beKSN9k3Fy1QuS4hhBCyqlDgtYAjRK3FcKmDlwi8IFKFoxSCSBpHS4esTOpY6GPPrYNXMSZBl6dOeq5KKTzw7DaOdOLozGgOVq3ajQwilaZoTrkwD619TsIiiSaTorkAYrSKNkYbNLIvzsEjhBBC5h4KvBaIB52P3i4Vb9mNdQ8ekLp49uIsyJRoTheykjp4SU+fN18Cz/TgFdypQgh0XAf9CV/7+d0BdnohPueaowDmowcvMGIsdfCmFZ5TO3gLFI2/LIO4Gw1Z0SWaDbx/UYOlo4QQQghpBwq8FnBEvQWQdlTy6Y26Bw9I+/Ds3QV2ieaUISt5gacdvPkp0Yy/Fjl4QFxSGoSTvXbtluk0znno28oIvKZ78CYOWdFfZ399RrEsg7jDBoVUkw5eU8E/hBBCCGkPCrwWqNuDV1R+CeQcvMFwmd4glHBFNmRl0vI77VrlBV5/Thy8qjEJQCzwBlFU/OQItDPWTaabT3oNm0SL1UEom0/RnHJMQhsC75GzO9jtTz7mIs8iuY1VmL65Bl5HmoLa3L4WQewTQgghqwoFXgsIIWotMLVDk+/BswXeQWGJpt2DJ5J9TFd+5+XGJMyD2AHskJVihee7YmLhoq9Z15+uj7FJBlbIiunBm/K89H6mL9Fs/vr8Tzd/FH94xxON7W9ZBEizYxL01wZCVujgEUIIIXMPBV4LuE69xLo6Dl4q8OyfU3ByPXh5kVgXLTKH5uDNiYOnL2NVieak56qv+1ri4M1VD57t4E0dsjJdD56+tZoue4ykwk4/zNzvTezT/rqoNDsmofl9zcnnP4QQQggpgAKvBeqmaOoFd15YZHrwkmHndmhE7ODF//aTksp8H19djIOXOIGuI+AITFz22DQjSzRdZ4ry1KREc44cvCCTopmW1k0TGmJCVibsVWwrOVEL8yb3uywCL3XwptuPUsp8SNKEPl+WEBtCCCFkmaHAa4E4RbN+yEplD15JiaZ23PTXcNLyO6UdvPRW6HiTi6amGR2y4k7cL6ive3eOylL1Odhz8IDp3LPpQ1aaS3S0GZQ42NOgL9PCl2g2NLvOvrRNXJFlEdCEEELIMkOB1wKOELUWw6Fx8BSUUvj2t30U77v3DHb7oZnNVlaiqXvSfHe6Ek2TomkJKN8tL3vsBRHed++ZiY41CVWDzgEdsjJpD15Souknc/DmYNFqp2ja5zPNuYXT9uCpdhb1+h5rsvSzrXM9bJoS1bbQnbdyT0IIIYS0AwVeC7hOvQXQwHLwBpHEJ564iNsePo/tXogrj3YBpCWaKu/gCX2spERzwv6xfA8eEDtaZaLpvfeewQ//0V14+sL+RMcbFzMHr8zBcwUG4YQpmjpkxZvuGjZJ0Rw8YLoFtX6PJx8IP/05FKHvsTZKNBddgDT1OpoWePrPwqJfX0IIIWSZocBrAUeIWg6CFhRhpExp3lPP72O3F+Cqo2sAUgfP3l8mRdMMOp+yB88SeJ0KB+/53QGAtHS0bWrNwUte+8EgGsulCnNjEibtY2ySwBJjkSU4p3PwpgtZaassL2jBwZNL4uDpt2ra12F/ZtHMoPPmy2oJIYQQ0iwUeC3gOKJWoIFZ4EplyjWfvLCHHcvB6xWUaA5CaVI0dcjKpAsuMwfPtUo0K5IpLx3EATCTlkXW5c4nLuCOR5+vFbKiz/Vbb/4Ibi+HpbcAACAASURBVL71kdrHyPfgTVrm2iT6tQSRygjOaRbUaQ/ehCErLZXlpT14ze1zaebgNTS7zn7Ppu3nA6zrO/tfFUIIIYSUQIHXAo4odyWe2+7h929/DEopBMliKZTSLHZPXzzA1kEwVKKZDVlRRvBMHbJS4uCVuT1b+0FyvHZXeL/+Dw/jV97/kBG2ZXPw7DEJT1/cx6mLB7WPMc89eIM56sFLUzQnPoVC2kzRXPQSwnZKNKfaFYBU2DFFkxBCCJlfKPBawBWidGH2d58+g//0dw/g3G4/MwdPl+ZJFS98T2z48F2RhqzkSjRdp+GQFadeyMpW4uBNKhbq0g8jDEJp9eAVb+e7cb+gUgq9IDKCuA7DDl7xawoiibPbvTHOfnLsOXgZgddAD14wYQ9eW3PwjIPHEs0hmnJNMyWaTYSsMEWTEEIImXso8FpAiLhEU4uTZ7YO8JlTlwAA+4N4BMJuLzTlV6FUQ4vvo2s+1ny3MEUzlCrtwWtoDp6dolmVTHnJCLx2F3iDSCGUyiwoRw06DyIFqdKexTrk5+CVXcN33PEkvu5XP3Qoi1p9XfMhKzPtwdNiYwHm4LUlRg+b1MGbbj/NO3jL4ZASQgghywwFXgtoN0wvqH7llofwI3/ySQDAbj8WINvWrLvYwcsLPA/rvls4Bw/AcMjKpCmauUHnQLbsMc+l/ThkpW0HbxBKhJEcGbKiEz97SZLmeA6eTtGsLtF89NwudnphI8EyvSDCb//jw6XXVwtre9B51bnVYdo5ePqDiqZFk76HmhQLyzKIu7k5eO304FHgEUIIIfMLBV4L6HJCvRh66sI+tvZi50s7eNuJEwbEIiu/+D7a9bDecQt78OxjaIEXTevg2YPOK3rwtIPX9kgBPQfOzMEruVN1yIoWX/sTOHhrfnXIytmdPoDx3MEy7nziIn7l/Z/F3U9dLHxeO7n9UCKU0pTgTnO59esahNPdI+3NwWtun8tSQtjcHLz4qyNYokkIIYSsChR4LeAYBy8t0dwbhFBKYc84eKnAi1M0swumo2s+1jMlmtnntUvoGgdv0hTNgpCVihJN3YM3qVioyyCUCKQ0ZaxlDp7uF+wH8fn2JurB0w5e8WvW/XfjuINl2CEqVc9rgdtxp5/RF1r7nAR9azVt2rQSsrIkJYRhQ06Z/nnPcRou0Zx+X4QQQghpBwq8FtBiRKq49PLZ7Z7pD0sdvPhrx3UQRtIsvteTRMfNNS/pwdNlbNlj6FRJIQQ8R0yRopmMSciErIjCEkIplXEeD8XBi1IHr3RMgi7RTITwWD14Zg6ekxyzeNX63Hbs4DVRojmqHy7fg9dN7odpFvrRiGOOoi3XJh2T0GAP3pI4TLJpgeeWBz8V8dCzO/jF9zwwVNbZlptLCCGEkOagwGsBV6Q9eM8l4g4Advsh9gZZB2/NdxBZJZovveoIAKsHbzCcomkfA4gXb5MuuPSaP+vguYViZ6cfmtfSdg9eEEkEUhnHoCpkJZIK+8l12p/AwasakyClwrnd5ko0Q6vHroiB5baFUergTXO5pw1ZkS314A1aGHQemXNtbJczQb+O6efgxV/dmrM5NX9//7P43dseG7rnKfAIIYSQ+YcCrwW0FpFK4ZmtNF5/vx9hv5/twVvz3ThFM1mRfsF1x+G7Alcc6cQ9eGUhK9Y75znOxKmW2olzcnPwihy8S/tpWemo4/WCaCrHq5+ErIwq0ewk7psWzOMc06RoVgw6f35vYBazvWB6UTtKbIWmhFMhlNK8vlkOOm8rRdOMBml0Dh4a3+csMK9j6jEJaQn2OPvqh8Xuaio8F/v6EkIIIcuMN+sTWEZMiqZUOL21bx4vcvDWOy4u7A1MuMZ33HQD3vSal+LERifXg5c9hsg5eJOWTBYOOvdEYY/YpQNb4FUf7yf+/B4oBbz1e79sovOK9+9YYxKKt9MOly553U96HcsGo9uYFE0zJmH4NT1nzb9rwsEbVS4ZWDPrIqmM+JxG4Jm+v3Cy829tDl5yPpyDN0xaojnlfkyJpjPW/dsr+btjSmAp8AghhJC5hQKvBRyrRNN28Pb6Ifb62R68NU87ePEifM138eKTR8y/dbBHfsGaKdF0nIkHneufc2s4eFsHg/TnRrhBT13YL3Xd6hBECgJq5JiEvIMnVVzmqINTqkgdPDfzvc25JEETaDZkJSgJqbFDWOIevETgNdKDN9k+mhIbeVrtwVtwAdJUWIy+tLGDV//n+iUBOPpziZYrtAkhhBAyBSzRbAFd7hhJhdNbB+bxvUGYhqzoHrxOLC70gsp307dkveOYT9LzJVG2ozVdyEqasqfxy0o0x3Dw9vrRxIIoksr8Z8YkjHTw0nPrDepdC/PaXQHXKe5jtB08u/zzLz55Cr9/+2O1jlN0zLIUTf14P4ygFKwevMkX+k314LVWotlCD96il2jq829qDl7cg1d/X/pezwtl2ZDwJIQQQkh7UOC1gBZfSimcvniAjUTE7fajdEyC7sFLHChdPtWxBV5FiabdMxeXaDbo4HnFc/C2xujB2+2HYwWe2BiXS8qk3BKlJZfawduxBsfvB2HhtnnsERGuU3wNdYImkC56w0jiF9/7IP7oY0/WOo5NMEJs6ccPBvWGsNfBiMqS4epVKJW6qE2XPZb1eU2DFkaT/j7Mkoee3cE9T28BaG4cgb4evjvemATj4DFFkxBCCFk4KPBaQJdPRkrhma0DvPyqTQDATi8wgm07ESTrifjTbpfnpkJGCzyl1NAn6U6mRFOMLJksww5h0HS8uOQz74KM5+CFE4esaBdLqXihXlXqmS/RBOqXUkaWuC1zQc/u9Mywcf3efeTR53F+t5+5HnWJRsyk06Wb+to1EbKi+zMncfDs266tFM0mJ24s8hy8t9zyEH7u3fcBSN/vaZ1IO0VznGtievBy782yOKSEEELIMkOB1wJakEQyEXhXHwWQ7edKHbxE4CULKrtEc63jQqn40/R8eVV2TIIzcciKdjqczBy8+BzyZYSXDgJ0PQdCFAeSaGQytmBSB892mgahLA1Ysc/VLtGsGyahRbHnOLHAK3Hwrr9sI7Pfv7rrFID4eoy70E3LJat78A5yAm8awRJG1ceswhZ1zZdotjAmwfSILZ4AGUQSfSOsGg5ZScYk1C3T7JeMsIhaKKslhBBCSLNQ4LWAFktb+wH2BpFx8M7aAs+agwfALOzyJZpA/Gm6lNm+u+yYhMkdPJ0kaTt4OrkxL/C29gc4seHDd5zKyP29pM9Qu4/jYjtNQSQrEzFTBy8ty6zv4CUjIkQskotEwdmdHm64PBZ4vUGEvX6IW+57Dl0vLnnbHdQrB9VogVdWLpkPHulUjHCog1Jq5DGrsBfyTS/qBy2WaC6iwJNJ3ynQ3DgCuwcv3l+9n+vlhKYmnTO4eNeXEEIIWRUo8FpAa6WnL8QjEm64fANrvoOzmX6ueHFrSjS1g+dlSzT1c1KpjLvnDI1JaK4Hzzh44bCDd3zdh+9Wh7roPkMgdQLGwU6YHETVDl53GgdPKniOgBDlPXhnt/u4+mg3HjofStz55EUcBBHe8AXXAsjOBqzD6DEJ2ce12J7UPbN/bJISTdsYbnp4uD6fVkJWFlCARLbAM3Pwptunfv88dzwnuKwHzziLTNEkhBBC5hYKvBbQYkn3aJ1Y97HZ9XBupze0bbeiRNPuz4uUyrh7Tm5MwqQJiZEpU8z24AHDgmBrP8CJ9Q48t/p4u30r8GSCMs1BlP7MIGy3B0+/V0U9eJFUOLfbx9XH1rDmOzgYRNjaj0dFvOKauOx2a0yBp69bmSDPl1FOOwfPLt0tS+6sQrZYojkoERHTsNAOnuW2NpVWaZdoxt/X+zn9AVTZoHM6eIQQQsj8QoHXArqk0O6jOtL1TImmFiWALeKGSyXXLAdPqWwAS9ZxK474r0PZHDyg2ME7tu7Ddx2TBlnEniXwJhkOPhhy8MoFnp8bdD7OMbWDB8TXNn8NL+4PEEmFKxMH7yCITFrn9ZetA8DYQStViZbawela94cZkzDhgtou3Z3kQwD7uI2naJpy1OrtLuwNcPOtj9QqV0yF0dSnd+hIpYZSKpsSeK4ReHUdvOL0XqZoEkIIIfMPBV4LuDmB1/VcHOl4JmTl5JGO2VaHrPSCCL4rMv1mmR68XImmrXncqXrwYhfLPq5f4uBdOgjiHjxXIKgovcwIvDF71PLHDUJZOgMPyDp4ehE7mYM3PCxei7fj6z7WOrHA0+6kDl6xh7/XoWomnX5Mj9UAgK4/3ZgEfbx1350oZEVZp9l02WNQMkw7zwceeA5vueUhnLp4ULkdkJaRLqIAiaQyjms6B2+6fer3TCfB1t1fPygp0WzR0SWEEEJIM1DgtYA2w3qD1MHb7HpmsX3yaNdsu95J5+DZAi5+LnX3IpkVeHaKZuyoTViiqVRmX0DqGuX759IevGExZLObEXjjn5ddSjjKwdMCb38Q4bINPz5mbQdPmt6kuAcve67arTu27sU9eIMIu70QriNwzbE1AOM7eLoMtEjg6Wu60fHMY9MOOo/MPt1Mj1ftn28zZKVmiqY+bp0+08Uu0UzPO1+qOc0+AcB1xu3BSwad50s0ZXv3AyGEEEKagQKvBXSKZurgOTjSTV2ZK2wHz7cdvOzboUv1ekmJpm+VaNpjDVxn8hJN28XKH9cuI4yS0QebXQ+eKzIi7I5Hn8dbbnnQfL83sHvwJnDwrOMGI0JW7L7EExud5JiT9uBlr6EObjm25mPNd9ELI+z0Amx2PZxIxOS4PXhVYxL067bvlWnHJGjRqj8sqFum+fj5PXz00fOZ4zYtmgY1Hbw0cGT08ZsqbZwFdg9e04POvXFLNEscPPv2YR8eIYQQMp9Q4LWAU9KDpzm5mTp4dp9dXuBp8REpVZ2i6TgTld8BcY+Wl1NQ+jj2PrVo2+x66LhOJpDkffeewe/d9rj5ftdK0ZyoB8/adz+sdvDsfrXj6z6EQO0B6/ZrL+rB0w7e0TU/7sEbRNjph9jseljzXXQ9ZwIHL+nBqyzRTO8V/fomLcHVP6fLfesGrfzuhx7FT/33T2fEV9OmWN05eOMM1zYhIAvo4EVSpXPmGnIi7UHn9vej6IUlg86tB5ikSQghhMwnFHgtoHWYLk/sJiWaGrtE0wi8QYSOmxUynpOW52m3SWsxW5PFISsTlmhKCTd33E6Bg6f76o4kDp4t/nqBxCCSZvtsD94kISvZQedVc/Ayg+F9BxuJEKuD7eC5BT14Opnz2Hos6A4CiZ1eiKNr8Xt5YsMfe0yCcfAKehi1+LLvFf1eTOqWaIFgHLyaYysOggj9UGYEQVsO3qj96nCVOtdALrCDF/fgZV9rU3Pw0h68ei6o/v0eGnRu3w8LeI0JIYSQVYACrwW0IOkVOHi+K3B83TfbrlllmF6JgxdKBalg5rXZz+l/T+zwyCIHL/7eLudLBZ4b9/xZz+lP+3U55rQpmkEu+bGyRNNOJPVdrHdc7E+SolnQg6dLNI+u+VjvuOgFcQ/esbX4/Tux3hk7ZEUL8eKQlbRfTtOZcg6eFgz6g4S6Tm8YxYEfbaZoDnJuVRmm7LKGNl1kB08lPXhKKROGMnWJpknR1KW+o39G99/ZP2++l+3dD4QQQghpBgq8FihM0UwE3kbHM+VyQL5EM+/gJSWaUkIpBddJhZ0YEbLyns+cwa0Pnh15rkU9eFpU9DMOXvxaNrsefMfJCEotZHW4yu7UAi/r4NUJWQHixMn1jmvCbUYxqgdvpxfCEcCRjos1z0EviLDTD7CZOHjH1/3xe/Ciih68SPfgDYesTDrIXgvKjVwPXhjJzEI+zyCSiCJlFvRCTO8mDR1DB3mMDFnRX2s4eIucoqkSUd1gkIne1Tg9eLr/DhgW35lzW8BrTAghhKwCFHgtYHrwBnp4ucBmEpxxpOOacjnPEWa2XS+QpT14YaQQqXjgtxaPbqYHT5jeHc1//rsH8Mu3PDTyXCM5nKJpQlYKHLyNjgffE1kHL9ClmZHZ9og1pH1cBmOErHiOMGMU1jzXzKurQyilKYMt6sHb7gU4uuZDCIF1PSbBKtE8vuGP34NXMSZBv+6iMQmTLvSDkh68X/37z+K73v6xip+TiXOclPg5TuMleSZFc4QzN04/WlqiOd25zQKZOPVhkwIvN+eyzv56lvAvG3TexLkRQgghpB0o8FrA0T14QYSu50AIkTp43dTB813HCIyDIMq4UUA62DyySjR1eqYtejxXZAaPb/cCnN46wGef2xkZOBJJNdSDZ0JWLKGlXbnNrheHuljH08fYMyWakekznEjgRfV78IQQ5nzXfAfrvjtRimZRD95OL8Sx9fh9MyErvdD0yB1fHxZ4Dz+3g3/xKx/Eo+d2C48ZVpZoDjt4WmxPOyYhn6J55lIPZ7Z6pT+nSzSNA+SKkUJsXIKwXr+cHKMHb5EHcevXad//Tc3B0w5enf1lHLzc9pkSTQo8QgghZC6hwGsBx+rB06JNi4IjHdc4NJ6b9tQNwgoHL+nLcUW6ULPHJOR74h48swMgXuTe98w2/uwTT+HN77y78FzjPrTscTtFDt4g14MX2j142XCV3X6YzMsTtfvhbDIlmpGCM+Iu7RqB5xqnrQ6hVEZEl/XgHe36Zt/9UBpXDwBOFAi8d3/qGTx+fg9/8OHHUUR1iWb82JFOQchKA4POgdQlDK1AjyIGiYOnj+s5or05eKN68MYIHInGEIPzhj5n28Fubg7eZA7e0JiEzKDzqU5tKvphhAfObM/uBAghhJA5hgKvBfRiKnbwdGnmcA9ex3UyASf5sJN8iqZjhazYfWkd18ksCu2Fz6dPbeH3b38cf3//c4XnWtSDl45JsB08qwfPzYqhfpCWZuqvRzpJ8uTUKZpRZQ8ekIog7eDVHZNQpwdPO3h2SMlRqwdvfxBletlufSjue/zLu04VJmxqMVPl4GVKNKd28Ip78KRUlcmrYSRN6AcQX+OmXbGg5hy8tERz9D5NieYiOnjJ68sKvCn3mXPwaoWsWA5e1aDzWYrov/nUGXzzb3147BJpQgghZBWgwGsBrZcOBpFZoOuyuyNdqwfPcvAADJVoZlM08wIPmZ/r5wTeZRs+rjraxV/f8wwePruL/UGUmV2nifvQxhuTEDuGBSWa/TRs5UjXw0ZnMoFni58gUvUFnpekaNY85qg5eLZbt+6n7409JgGAWWSe3e7h3tPb+OYvegF6gcSb/+xufONv3o5/evxC+nrkcBmeRj+20S0QeE314CVlkaGUlQ6e/jl9D3iOM5Gb9IEHnsNz28WloP2ac/DGCU5ZZAdPtuHgaQfWrZ/Gan9AMjzofD5CVrYPAoRS1f4whxBCCFklKPBaQAjbwcuVaHY9K2TFMSWCAIZKNO0UTani3j4TsmKJsq7nGpcPiAXeK689hi+8/jjueXrLbLfXH14MFaZousMpmvuJwFv33WQOXkHIiu7BG4TY7I4XeGIzsMTjIJIYoe/MdVvvuFj3vcbm4O1YIxHWLVdNv5fHknEXepzCBx86BwD4kde+FF/1kivwwYfO4b5ntvEp6z2oHJOQXO9MiaabhKxM2YO3lnPwIlktmPR2gygJCvLE2OcgpcKb3vFJ/PHHnxp6TillhMwoEaPPs06JpjTbNp/62TZRgfiffg5e/HWsHrwKgZlx8GYo8Ba515IQQghpGwq8FnALevCOJK7MRsfDhh8v4DtetkQzPyYh4+DpEk13eEyC7bhFUuGh53YSgXcis7+dfnHJYJnAy5doHum4cBwB3ymeg7fbT0NWYiHr1XbTbGwHQ5emVqFffzwmwan9qX4mRbOsB28tW6IJIO3B2+gAgBmV8I8PnsW1x9fwudccxe98z5fi7/+P1wDIzhXTzpgOGMmeT/xY0Ry8Scck5HvwUoE3ysGTybnHXydJ0QySyP+DQTj0XCYpckTp5TiDztuc29c2xQ5eM/vUv+MKo3fYz/z+Fe8v/+/DZpHnHRJCCCFtQ4HXAlqQBJEadvA6LtY6qahwrQSRUgcvsko0zZiEdDtb4D3x/B56gcTnXnMUX3j9cQDA5193DEB2Pp0mLBB4jiPgOWKoRNMMa/dE4Rw8O2Rlc83Dul9fbNnk3a2qMQlAKkjXPAcbY4jKfA+ePWpCSoXdQWhcOlvg2SmaQFyi+df3nMYt9z+L13/+tRBC4LIjHbzsqk04Ir9gHt2DZ6doatE/uYNX3IMXqVEOXq5E0xVjh2ron7Vff/65jjtaOI6zmLfPcdHKNPXrq3LQxsX04OkSzRq7q12iOUuBJ3Wp8WK9x4QQQshh4I3ehIyLnfrYyfXgbXQ9bCQleL6bd/CqevBi4WV68Ap69/pRZBI0X3ntMXzO1Ufx41/7Mrzsqk28+Z33YLc3LPAiqYZ68PS5ZBy8QToewHMcU0amlMrMwQsiiUEosdmJX+e0g84BjBGy4sbBLkEEpVTleAVAJ4imPXj2YnGnH0Ip4NhaOiZBY3rwEoH3Ox98FPc8vYUvf9Hl+OnXvcJsJ4RA13MzC3Z9jMIevLBI4DlwHTGxWNFCfD255/S5RIm7VnaddL+m3YM3toOXHNsO7dDo/a75DrYL7ksbfanqDTqfj5THSdC3X7shK+M5ePkPFqTSbrdqfGzGOOj7uiooiBBCCFlV6OC1gC1IdIrmRsfFD/4PL8bXv+pqaw5eNmQlX6IphIAj4kVZ7OChMEVTjwkYhBIX9gcAgKuPraHjOfjJr38Fbrh8A0AsWvIUOXhALJrKHLyO55gFVj+3jR3GsjbGTDqbQc7xGSXUbIGnr22Ra5Qn24MXL1offHYbX/jzt5gk0qIePC3wrjrWxYkNH4+e28XrPu8a/P6/uSnj9AFA13dMyiiQCqdiBy++pkesY7lO7No2V6KZOB9mgVy8X90HqYWo740fspI6eMP3gH79+rpWOZS6RLPOWn5eUh4nQRoHL71eU/fgJdfMlGjWGZNg3a/5axhJZT6Imm0PXtpLSgghhJAsdPBaoCgZUwiB//CNr8o8PsrBA2LnJJuiGW/jlPTgaTHRtVMfE2G2U+LgdXKiRO/Tdpn2+5HpI/ScNGTFdmf2BmFmIPp6p/7IAptBJDMBLbVLNH3HlCLuD6IhsZUnOwfPQRhJPHZuD9u9ELc+GI87OFro4MWib6Pj4c7/8D/CdUSpCO3mEk5NaVnhHDydopn+WnqJaztpiWZZD54Wa/E1KPq5rIPnO8IEl4wS3Jp8H59N3zh48cEjpeCgeL8mUGPJe/AK5+BN+Rr0+2xKfccOWcmdo1TwXYGDIN73xx57Hlcc6eDlVx+d6jzHJTL3LxUeIYQQkocOXgvYgqTrFV9inUbpjhB4rhPH90uJRODpx9Nt7MHkenFmH1cLkqISzVIHz3VMpD6QjD6wSkv18HW7BHOvH5mkziNdDxu+i/2CgI1RDEKZcczGKdHUQqZOaWjs4KX9kHEgSPxzn3giHm2Q9uCl13PTFmCuUyl48iWaWvTo4Bwb04OXd/AcMbFToZ2O9VwPnhZ+ZSJIJ3raPXhV2xfRr+jBMw6eFngV+9VP1RE79jaLNgvPhKxE5QJr0n3q+7zWoHO7B69gDp7+fYukwr//q8/gt299ZLqTnIBR9y8hhBCyylDgtUCRu5Zn3XcTB8+p3FYP4DYlmmI4RVOLuX6QCryOpQA3ExdqtyBFU1aVaEZZd86ErLhpiEwvI/BSB0/P+5t0Dp7tmNV18NZ914wDqHNcewagm/TgaWH4mdOXAGAoRbPrOaXvaRGxg2eVvFkL0iDnPujrvea7ZjSE5zhwxOS9RuHQHLzscPGy0s8gX6Kpy/LGKBmsKtHU++0m51UlPPRzddbyGQdv4Uo046+NzsHLjUmocxsVOc7me6Uy90I/kDOZRReNKDEmhBBCVhkKvBZwCsRXno1OLPDczBy8YSXjuiKZg5cddO4WlWhGEv0wnr1nC8CNRDCM4+D5rjAuDpBL0XR1dL80IxKA2OXby5VoThayojKjApwRCi918JzUwash8KIom6JpD07WAsf04CX71W5oXeIevOzg9qJ/A+noBN91jGh1XQGvRtJkGaZEs5PrwStwQH70T+7CL733QSiljPg0JZpmUHb9YxeV8Wr0fvUA+WoHr36Jpn1+i+bgFZVoTqtR82MSaoWsVKRoSqsHTymFIJJD9/FhQAePEEIIKYc9eC1Q1IOX54tvPIHrT6xnevBsNy99TFgpmlbIil2iaYWs9AM51HvmOAKbHa8wZCWyXCybvIO324+HlwNp5HoQpgmax9bi8QR2yMq67yKI4kVgUflpGYNQZgXeiBJNve+u55pSSj2gu4pMiqbjIJJqKBRGl2hqgaQdvbrkSzQzDl4ogW66bRBJE6TTcePePc8RcMTkJZr5HjwtHtKY+XTHD5zZxn4/TNI148f0PTBOCqNmUNGDly/RrBKO+lzrCLaFDlkpLNGcNmQl24M39aBzpcy+IhnfX0WBQW3DMQmEEEJIORR4LWDrkW5RggWAX/2OLwaQTVMsEoOmBy/n4JWGrCQOXp7NNW+CHrykXyuS6AUyTdHUJZoyLc86udnFpYMA5/fiFM/LNjpGpB0E0XgCL8qK1JElmlYPnr7eRa5RnvwcPCCd5acxJZrepAIvW6JpL0jzC2NbCPueA/R1D94Uc/B0KaQXj1tIB50POyD9QGIQZQegGwfPy5Zohsl2VUE2QUWJZlHIShnGwavTg6eU+Z1ZJHdHKUtUNzomIf46aQ9eXrvlUzSDUA6l3h4GHHROCCGElMMSzRao4+Bpsimaw0rGpGjK0QKvnzh4doKmZrPrlaZoFjl4a36agLmffNUhK8bBi1KBd/mRDnb7IU5fPIDvClx1tGsW770x+/CCSKLjpQmjo1Ibu1aJZte6FqOwHTxdKmvPZNN9kkDsgnY8JxOwUod8imYo0/7C/Cy8QSSNG6vvBVc0MybBdUVcdpsTeHaaZz9ZrNvnZadoAqnQpsY26gAAIABJREFU/M0PPIx/9baPVh67X+HgpSWaNUJWxpyDlw6HH7n53JAR2m0MOh9zDp52wotKNPXfG5mU8s7CRWMPHiGEEFIOBV4L1OnB0whLtFWmaCok5XuOeTx/DJ2iWeQaHl3zTACKTRilSZI29pgCu+zSPs8wSks0r9jsoB9KPHVhD9ceX4fjiIyDNw6DMBY6qZit3j7j4Pla4I2XouknX3f7IU5uxu5j3q1b94cfG0XXczNuYhSpoX44TRgp45T5bhyu4jgCrismXujrxbfnCPhuWnZb5IAMwgj9UGZ6L9MUzWyv3KmLBziz1as8tglZKXBT9WuvE7Kiz7XONcg4TAtUommfa7YHr6GQlbHGJETYSD7MKS7RtBy8aDYlmuzBI4QQQsqhwGuBOimaNlUCL+3BU/FiX+hjpNt03LS/qrxE0y/swZOq2MGzA1L2rGTM+Dzj7XWoCwBcsRk3kz383C6uO7Ee78NPZ9KNg3bwjHs2wsG79vgarjrahe86aYlmLQdPmoWvfg92ewE2Oh4+5+qjQ2Lu+LqPy490xnotXT9bohlYDl5xiWZ8Hh0rYdUVYuKFrP453ddnxiTkwlaA1MHLlGiaFM2sQOhHcsiBzJPOwStI0RzHwTNitPJwZj+dnBhdBGwd1eiYBEvgx8epU6KZ9sDmr6GUsHrw4jLYmZRoyuy4jzJ+8b0P4Pdue+wwTokQQgiZG9iD1wLZOXjVw7aBePE1QEmKppOkaMryQefatYoFniwUeEe7Hk5f3B96PJSqMKVy3U9HHOwms+02Cx28ROAlwufx83v4ohtOAEA6smBMBy+IYpdAi69RAu/7vvKF+PabbgCQupl1otszPXjJsXb7IdZ9Fz/y2pfi0kF2rMRbv/dLcXKzO7SfKooGna9Z75fNwOrB63iWg+lMLvC0kPOcWDDrpM58X5tSKi3RLHDwTIpm8nP9QBYOa8+8njpz8Dqje8O0SKlXomm7VYsj8LJOavMlmmkP3uifiR087axmnwtleo/q93UmISvJeY36vfjQQ+dw7fE1/OBrXnIIZ0UIIYTMBxR4LTBOD569fVmJZjwHT5doxo9nXEI3LUvsB8Ulmpvd4hLNsh68Ygcv6cFz9Bw8mZZoJgIvlMo4eBtjjCywGYQymRGoe/Cqt/dcB5smSXPCHrzk604vxFrHxdd/3jVD23/eC47Xfg2afIpmKJUpf8u7D0GkrB48x0r4nMbBkxBJMqfvCeMO6WPrxbmdeFkUsqLdRH0e/TAaubAPavTg6fCayhTN5HTGLtFcIAevrERz2lEPQ3Pwajp467pE0zq+UvHfId/6ewMMlxofBtrBG/UeR1JhBqdHCCGEzBSWaLZA0RDyKryqEk03TdGMExWzggTIpmj2wqg4ZKUsRTOShSma9pBye7YdkCYq2iErJ4+mztZ1l62bfQATCDwTslKvRNNG93SNStGUySgAI66TY+30QjObrQm6nmPmiunxA6UlmqGVoumKjIM5aT9ZYIlY30pG1Qv3VLClYsw+r36uRFNvr0s5q0r+7JEMYe619o2D12yKZqQWtETTujzNpmgmDu4YrmY/jMyHM/b7os9F3wuzdPDSEuPqY+uAKkIIIWSVoMBrgaIAlOrtU9em6DndgyesEk1b82TGJATFJZqbXQ97g2ho0Vvq4Pmx8xRJhb1BLPB02ZYWQ4EVsmL3pl2vHbzEBXh+rz/qEmQIIomOK2qHrNikDl61qLTDRwCrBy8p0WyKuAdPu2aJa6VDVnLOVhBJ+F4qxuxAnWl68PRr67iOOQf9+kNLsMVfs86cPse8QEgX9xUCL5fQaaP3262VoqlLNEs3MSgrBGShSjRtB6+FOXj6HqizO7sHzz6+fo+MgxfM0sGrJ/qDSC6U0CeEEEKagAKvBbI9ePUdvI5X0IMn4kWWKdFMNrFFpOcICJGGnhTNJtOBIfkyzVAqMyLARoucXhAN9eDpxX4YxY6h7wocW/PNz2oH70VXbODFJ4/g929/fMjBqWIQ6pCVej14NvFg8NElmmn4iJN5Tbv90LhKTdD1XISJg6Vdh42KMQnatbTHRExTohlGKg1rsfaT/5p18IpCVrI9XHXcG9uJyr8fer9r3mgxpp+rO+hci+RFWtiX9+BNt9+0RLO+6O2HkfkdsK+5/tlOrgx6nlM04xLNyS/iXj8cuwKBEEIImTUUeC0waYqmVzCuwHOcpAdPxTPRCsoWhRBmMHlpyEqJwKtK0QTigJSyMQmDpERzzXPNcwBwzfG1+NxdB//uda/Aw2d38d8/eWrkddDogd/6uoyag2cjhBjqeytCO1l5By8OQWlS4KXXSi9Ky8YkBNYcvMyYCGfyMQmRlRTqOcIc08zBk3qUQWTO016w50NW0hLNePuqoJWswIsKn1svSWvMvobk64hrkO8RWyQHzy51ta/V9GMSsg5eHcGYTdFMH9fvUWcOQlbqlu0G0XQlmj/+p3fj3//VZyb+eUIIIWQWtCrwhBCvE0I8JIR4RAjxMyXbfIcQ4n4hxH1CiD9p83wOi+wcvBopmu7oOXiR1CWaw8cAYiHZD8vn4G12Y4ct34cXylg45lm3AlL2+iGEsEo0jYMXl2h2fdeMULjqaDdz/G/4vGtw0wsvw2994GHz2F5B2ItGKWVSNNMxCaWbF7Lmp31vZdjjA4CsuG60RFO7HUFaKpYKvHyJpsqUaJoRDlOMSbB78DzXGengKZXtmUwFXnGJZtWohEwvX5B/rRKOGBaORdRdzKc9YnqflZvPFVFG4DWboqlDdururx8Uz8HT5zhcojnLHrxRDp6cysE7c6mH57ar5z0SQggh80ZrAk8I4QK4GcDrAbwKwHcLIV6V2+blAP4vAF+jlPo8AP97W+dzmEyaollUoum5AqGUUAq5MQnZ7bpePMS6H5TNwdMOXhr9nwaNFAw6txy83X6IIx3POGl6gRckx1vzHRxJFoS6PFMjhMDXvfJqPHOph71+iI88ch5f/Avvx9mSRZMWDJkxAWM4eMBwcmURpgfPcrc0zfbgpXP5dJlq9Ry8+Nped9k6rjkWO6FOMgtxEqJIZYJ59DH1ojffgwdkBbh+P/IpmoM6JZq2wMuXaCaBMvrDhTolmqPcrLRHjCWa6c/H41X07V1rDl4o0fUdCJF9X2Tud8buw5zWaRwXfb1GCVadQDwpgzAaOQ6EEEIImTfaHJPw5QAeUUo9BgBCiHcCeCOA+61tfhDAzUqpiwCglDrb4vkcGpP24FU5eHGKpjUmIafwup6blmgWpWgmJZQ7iYP3i+99ABt+tqfOJu/gbVolmHoBHUiFXtLzt9FxIQTMiASba47HCZvPbvdw3zOXEEQKz233cVUiYGx0CWHHdeBpB2/MjyHsYJMy8g6e3YfYbA9eGvqi3199bYfm4Fkpmj/z+s9NExAdMfEw6XgUhJPZj1IqdfCirCMHZMt48w5e3vGbuEQzSUq1S2PLSN3G0k0ApIv9RSzRtMMgGw1ZSXp39YckI0InoVQ8uLzruUPOsf4wQJdo2rMmg0gVfkDVFiYsaIT4CpIZopPSDyWCUReNEEKWCKUUvuHXb8OPvPZl+NYvuW7Wp0MmpE2Bdx2Ap63vTwH4itw2nwMAQoiPAHAB/LxS6n35HQkh3gTgTQBw4403tnKyTSJEHHqiVF0HrzxF00vcm0jGn8SXjQ7oeA4OBhFCqQpLNO0evDOXDvB7tz2Ga4+vJ8ev7sHb6YXGAbTPM0zm4K35DoQQeMnJI/jiZMi5zTXH4uM8e6mHZ7Zi525/UFymGViCIp2DN66D54ydoulbKrLZHrzUwROIj1WnB8++F1xn8jEJmR4818F+Lkk1NIItvV6FDl5yPvo09PZVJZqjHLyu55gPKqqEjH5q1DXIpzwukoNnv35bGE+rUbWDp3+FRglG/T6t+U7S+2ntK5+imZnvKNE5xJbuuimaurx9UgahpINHCFkppAI++9wuHju/N+tTIVMw60HnHoCXA3gtgOsB3CaE+AKl1Ja9kVLq7QDeDgA33XTTQvzfVs8uq9WDZxy8ghTNxMFTKhY6WtjlNVnHdbDdi8svy8YkALGD985/ehpSAae3DjLHtzEOXlKiaTt4nlWiqUNWAOAffuKfF76+a5PQlTOXejhz6cDst4i0RNM15zVJiWZvxBw87VzZowg0bfXg6dejBWR+hlevZMSFM0UPXmiNSYg/LMj2JBlHzrpee4U9eIloUsq4PEB1iaY9BiLfg5cv0axy5/Q5jizR1CmP1rkuCqWDzsd4Dbc+eBYX9gb4ti+7Pv15qUs064WsaNHWcR04YkQPnnWeQaiADg6NfIlxEbqfdxoXNB86RAghy07d/+eS+abNj1xPA7jB+v765DGbUwDerZQKlFKPA/gsYsG38OiF6zg9eMUOXjoHzy7RzLtuHc/BdlJ+WdWDd3F/gHd+4qnMNpUO3iB28I6uFZRoRioWeIlgiZ3L4X3pVM3ntlMHryx63C4JTAd9F25aSh0HTy8Q0wCSlko0/bREU//R1GE1WadG4dxuPzMwXjP9mARL4EUq5+Cl4xE0o0o04/sx3X8ZWQcv+34ESYmmNk6bCFnRw8JNIMwiOXh2D96EJZp//PGn8Hu3P5bdb65Es24fo+eIoRJNk6LpDZdoVjm5baDvu+rezfjrNA5eP6DAI4SsFmY0EQXeQtOmwPsEgJcLIV4shOgA+C4A785t8y7E7h2EECcRl2w+hiVA65wme/CyISvDAm9HO3gFDtRmx4MjgF99/2fx3HYfP/Sal2SOkcfuwdvt5wSeYzt4cYlmFWu+ixMbPs5cOjAO3n6JwAuskJWyctRRdH1nyDHKEyXCxg4g0TTr4Ol5gul8uTRkJf3jud0LMQglrioQeM40Ak+qzKw/Xe6rMaEp0YgSTWuOmi0GK0s0R8zBGzdkZWSJZs5hWqwSzfTfk4asSDVcjiiVguOkISuj9mf3pubHc8icgM44eJHE+d0+nnz+cEp6zJiPig8Y8oFCk2CPN3n2Ug+ffPLixPsihJBFIBV4Mz4RMhWtCTylVAjgxwDcAuABAH+ulLpPCPELQohvSTa7BcDzQoj7AdwK4KeUUs+3dU6HSZqMOa2DFycfyqREs3RMguuYAJUiweU4Ar/0P38hvv+rX4Sf+oZX4AdHCTw7RbOXC1nxdA9eHLJSJCjzXHNsDU8+v4/zuwMAwP6oEk3XsXrwRu4+w1gpmi334K0VOHhrBSma53ZiZ/PKAoE3zZiESEqzIHcdZ6gnyfTgBdUOnmc5eIPcwr6MIJI40tE9iPk5eCoz669OyMooR8704NUYnj5vFKVoOmK8Epmiod66RFPUENL287EoFJlrnhfQ+fvgLe97CD/0jk/WPt9pSHvwyu+/sOZ9U/rzUTzaRIvI373t0UN7fYQQMivqphST+abVHjyl1HsAvCf32M9a/1YAfiL5b6nQAqyWg5csnjslDp5eYDgCJlIynyyZcfBK+v6+49U3ZL6/7sQ6Tm8dVPbg9UwPnp+eb7L9IJLoDdIevCquOb6Gu6xPv3tlDl6YLiLTEs0WQlZMD16Bg9doimYasqLT+Lp+3N9ki6OzO30AJQJvipAVuwfPTz4sKHLwbEG83x/uweu4toOXPl/loPRDiaNrPvYG0XAPninRrOPgZb+Wb5fcO0Y0Vm8/TxSFrHiuMzL1Mr+PvJhJSzSHj1OEcfCEGLrv8iE22RRNiYv7A1zcH9Q/4Smo04On+2wn/d3RHzbp39P9fmT+xhJCyLKi/6xS3y02hxd7tmLoBVWRaMtjUjRL5uDpBYYrhClpyw8n73qOCRapIyoB4GVXbWaOb6MF3m4/jAVeYYqmQi8cXaIJxEEr29aQ9bISzUFhiWadV5PS9eqPSUgTJlsOWbEcPM8R8F0nU954LhF4RSWariOmcCHSHjxd7jsyRdNKOM3PC5Ry2LkpI4ikKe0dTtGMYgdvnBLNug7eAoas2K+/n1xTP1ciOYpIDs98S+fg6R680fsALAfP2j4dQ1FUohmX7pb11jZNWEO86Q9UJp1yoO9z/TsQyHgMzSL1dhJCyLjIKasfyHxAgdcSjiPQ8ZxaEf9pCEaxg6eFgOOkwSP5/dqloHWSO4FU4BU5eJ7roOM6OL8bC4+jVommm/T0mBTNWiWa2fl4+0HxmAQ7tXFyB88d2YOnF22tp2jqEs0gjVv3nPjaarcSSAXelUeHZwNONybBStF048Ae2/WIknvLvl661NdOdTWDznM9eJWDzkNpPhgYDllRuTl45a+hbsP3UInmAv3PqahE03OdsXogikYCSBX/3ag7JsH+EMIR2Wuo79+ikBX9t6A34bzGccnPcazcZsL7IJ8UGxTMjCSEkGUjMv/PnfGJkKmgwGsJVwh0a7h3gN2DVyC0HMcsMISVhleUoqkpGnRexMuNg1csoNZ8xwgPO2QlPlfHEnijj6eHnQOxgCr7pD8NWUln/o07B2/Nr5GiWdGDt95p7tfCLtHUiZVekhCaL9HseA6OrQ1XTTtCVC5kqwhlOjw9TtGUhQ6e7SbqGYW2cLeTKW0xmJ/lZzOI4hJNoGxMQuosVffgxV/r9o8tZsjK8Ln6rhirB0+q4ZEAcQ8eao9J0P9jdx1hHN/8ORaOSYhid2sQykO57vq+rSrRnDZkpZ8XeMn3ZSNeCCFkGWCK5nJAgdcSQojaQisu2SseMeA6wiyiXSGs2XDZ7boZB6+mwLv6KIDicBcg7kXTAm+zQOAdBBGkQs0evNjBu+JIByc2/NECz3VLX+soun6dkJVciqYlrpsddJ6WaKauoTACWXNup4+rjnYL7wGvIQfPdcpTNO3rtdeP4IjsfeRZoslO3Kyeg6ew7scuXdGgc9vBqyzRNE5M9WvVL6ujA2EW6H9ORa/Nc5yxSzQLUzTHGHSur7Uu67R3ly+BtT9EGYSpszvqw5Um0OEqVa/HjFKoEIFv+9Cj+JOPP1X4nH49ej/6b0aPAo8QssTosnbOwVtsKPBawnXq9d/F24pSkWV/iu4IYUIpilI0NXVLNL/0xhP45X/1hXjtK64sfH7dd034h52iCcTuQpraWS9FEwBecGId6x23PEVTl2h6YsqQleEV8z89fgFPX9gHMOzg2WWqbQ061wtF33GGevDO7vQKA1YAPSZhsuMHVg+e7yZz8Kw/2mmKZvp+7PbDeISBlWLqWUIs6+BVlGhGEh3PTfpDi+bguSYVts4cvNopmu7ilWgWCRXPFeOVaKphhy4OWbF78EZcQ8vBc5ziQedeyZgEfQ8dRh9eVMPB089VCf2/+dQzeO+9Zwqfs3vwlFIYJL+/dPDIovD22x7FG37j9lmfBlkwWKK5HFDgtYQjRK3xAUAaulGEXT5pL7SdnK3VmcDBE0LgO266oVSgrXc8I/DyJZreiLEMefSw82uPr2GjU16iqcNXNnxvcgfPi8cB2OJDKYUf+G+fwO/e9igAZNw0ICfwGkzR9NzUwbJn73U8J5NAqR28IlynOg6+iqyDNzwmwXbw9DXYSwSePYcwLfHL9+BVlGgmZZhFgrufPFcnvl//z2aUIzdUojmnnz4+/NwOwpwwLhKjvjuegydlQYmmUnCcMUo0rQ+T8uM59Dm6Iu7Pywt9/R4fRh9enR487biNGsExKDnfTFqsVKZEkw4eWRSeurCPpy/uz/o0yIJhQlbm9P+hpB4UeC3hCDGGg+cU9t8BWeGhhw8Dwymak/TgjWLdd8zixx6TAMQJf1WD1fMcW/NwxZEOXnLlJtZ91/R55dEz2I50XVMWOG4Pnt33prm4H2C7F2I3EaWRFXhifwXqlZyOdz5OpkTTd+OS3HwPXpmD5yXCbBQ//I5P4i23PJh5LJTSXEffFQhlcQ9ePNLAM4/5buqg6sh8IC4lrB2yEkl0PSeZSzjs4HW9NEWz6vXp/8csQ8jK87t9vO43bsd7730WQHzOSg3PrwPi3/dx/v+aL78F7BTN9Psq9OcIRYPO7SHoriPQC+1S3XR8xmE4eHV68OqUaMYlx8X3sC38wkixRJMsHPHfl1mfBVk0OOh8OaDAawnHqS+0XnjFBl54xZHC52wHzxECX3LDCfyzl53EZUeygssuy2xKoNhO1lAPnufg1MUDAMDx9ey5FCGEwLt+9GvwY1/7Mqx3PByUpFymAs928MYUeCa5Ml2IPZWUZvatsitguAeva81mawrtYNmz9+wevH4YYWs/wFUFCZoAhnqhyvjUqS185JHnM49FMjsmQaqsKNPO4CCUOGa9j/kSTX1JotwcvFFjEjqug64/7OANkufaGHRuevDm8P9O270QkVTYOog/HPnWmz+Cm299ZOj9NSmW4zp4BSmabmbQefU+0hJNfd8Nl2hqR9c+tThw6fAEUJ1BvGbQecVrrnLwbOEXSGlKNHsjEnoJmReCaNjVJ2QU+u8re/AWGwq8lnDHcPD+t697Of7i33514XNeRuABn3/dcfzRD3zFUJ9dOw5eKuqGSjQdgdNbB+h4Dr76pVfU2t8Nl29gs+thw3dxUOLg7fVDeE5c1pf24I133mmwSboQe/L5PQDp4nNoDl5ykCbLM9Pzicc2mJlypgcv/v78bjwcuszBc5203KyK3V6IJ5LXqQkiNVSGal8Xew7ekU521qFvjZAwA8lzC+LRJZpOLHALUzTrDjrXJZqlm2S2S0s0q7efBaavKxEPT1/cx6mLB0PCLJ1DN0bISoETKJXKCPSRPXiVJZrxV+3g2cQlmvHv1mGErNRx8PSHF1WlumGulNsm7+CZFM1DmvVHyLQUBS8RMgp9y/DDgcWGAq8lnDFSNKuwh5BXOUu2mKwrLEdhix178Q+ki+h/8YorTRR+XTY6bumg871kqLoQwrz2cR21ohJNHa6iP33Pp2gagddgwIo5n2Rsg17Ue65I5uDF31cNOQfi12+nWn3q6a2hhbqUCruDEFv7Abb2B+Zx28HTpZq22Iqs2V5rvmM+KPDddBGvF/tAUQ9e9Ry8TkWJZidTolm6m9r9AIsQspIKPGW+FpVWatdtnJdQ2IMns4POR10TaRw8XaKZPpf+zgyXiPcCacT+waBdh8t+DVW9qUHdEs3SHrzsfW5KNA9BwBLSBCFLNMkEsERzOaDAawnHqe/gVZF18CoEnpfOOvOaEniJQD3ScYc+sdeL6G/+oheMvd+1jltaxrXTD42Y9K0SwXGwRxNo0hLNnINnlS8CLQk8XaJpHdP30h68dMh5WQ9eOibhjseexxtv/gg+/viFzDZ7g9D8j/zx86mLF0pl7ofUwcuGRwCx6IvFWLKtm3VQ7VLKjANYosykVEkvnzMUsiKlQhCpxMGLH6tyWUyiV01xMs8hK3rEhL7uQSQRRnJImLlJieY4JTKRUsjrnbQHL0nRHLWPXJCKfc1T8TdcxrzbD8y/2y7RDDMCb3QPXuW9VVWimRN4qYClwCOLQSSH/7YQMoo6JfBk/qHAa4kT6z6u2CxesI9DtgevfDst8OomaNZBi518/x0QOzwbHRdf+7lXjb3fDX+Eg5eMZHCtkI9xSHvw7BLNvIOX7cETSZBIayWa1gBoL0mo1ALv7E4PAEp78HSpnFIKdz+1BQD47HM7mW10oimQvlYg/h98fhSEvXBNUzQjdJORBgCSFM00sdUupbR/flBSB6n7lzrecA+e/Zxbw1nST40qNdJPd7z57cHT96QWxmEihPX/SO3kWCdXIjmKSBaVaMbvn3DS76swc/D0oPNMyEr81Q7d0exa91/bYwSKUmCL0I6bUuVCORwjZEV/fxgpoWT+2Nof4D2fKR6pMa8EUXGAEyFVGGHHW2ehocBribd+75fh//7GV029H8+t5+DphXnd0Qx1WE+ctPwMPAD4l6+6Bj/y2pdiozP83Cg2Oi4Ogqhw0bXXj3CkG78G34rpH4e1qhLNIQcv/RVwHdGigxcZQadDVrQ4urgXl1Tmg3M0qbgC7v3/2XvzYGmyuzrw3Nxqecu3L71vUiO6JRBIiFUyss1mZgCv7A7ssPHYxhgcM45h7LDDeJkwYRsCg7cxxkOMGeNwWINtMGCDQOxIaEM7LXVL3V8vX3/b26sqlzt/3Pzd+7u5VGXtL+vdE/HF996relmZWfneuyfP+Z1zYw+ArdIBJpym+FjCZvB8XVBdNYOnUi1JdY58oc8NqTkApWjmVRaRX2vRNIX1ZYsmPdYJmoWsrNKiuXcc4+t/5Nf0eV40hvmxx5SemdE/9TjdqCEFb1qLZilFM5OaLALNe/ACr77oXNUuqK/Rz8sBu/6Wr+CVr9/K56WTiWDRcsxhK92Zvm4HTsE7k/gvH3gRf+nfvRd7J/HkJ58SUIqmC8twmAb0K9YpeO2GI3hLwpWdDs71p5tNq0IxRbMOtDBfjoJXPo6/+OVP4Lv+4Gtn2m438iFldRrd4TDRr+czNWMaaAWPhT68tK9UsmGNggcoS+hSFLxQhYwYAiIQBQKjfP8OhomeVatCwEjQh15UxOO5AsHjCh4PWkn4DF6FRZOnaHZCX98g4BZNwWbw0nxBrI7Bq7VoktpBtk+upuoy+4qQlb2TGF/3I7+GT756qJ9fnAf45Y/ftIjlh27s4at/6F3YzxdeRExnuXP9q8+8ig+8sIePvLg/9fc2AQ9ZIctfwpQ3TvDEDCErQNlWOU1NQsIUvHqLprF89vOfF67gLZvgcRvqeAWPEbw6BS/NGs7gSb09V5NwNkE35MbNHZ820DXr1ukO08AVnW8GHME75bBm8Ma8W0RqFkrwIrWt3QqL5jzo5ySiysp1OEywTQoeIxjTQIes5KTihbsnkFItRvUMHgWeFHoG60rf5wFZNDmp3IoCbVM9GibYqVBJCUSC7hyN8PwdVU3BbZgAdCfhdiewyF9aMYPHiTVX8CLfKHg8ZMX3zD7I3KLZCXxLhSyCyIsmeBXJm5ZFM/+D8vydY3zwhT383gtGQdOl1pnEs7eO8B0//m780sdu6sf3ckndAAAgAElEQVQ/8tI+PvbyAW7cU+cm8AWEmE3B+41P3s7Px3IW8ZrgZaZXjey3gFEfiZRNszBLK+66kkWzadE5LzP3PVFph+QWTfp5ObQUvOUugLmC18SiCaA0m8i/vzZFk33dStF0BO9MQupFb3tWvU2SZB0cishaeK07lOEI3imHlaI5VsFTC606FWgWjLNozgOydVaVnR+xkBV/RosmkVyyY1LAymuv7ZRn8JgFNvC9pVo0yTIWeB62u4FWPQ4HCbbGnGOyqv7GJ28BAD7nwXP4zJ1jSz2jBfbT9+/iOUb+Ej6D51cpeGwGz0rRZDN4XMHLVA9eFHgIPTFRwQsrLJr8MV6grvZX7Q+3/PHI5qP861yxpGMgwuyJfG5xhj9Ov6kJ3nJIiu5hZIpQnBp1lwg2qWRT9eDpOomigmeCipraXKtenz7mhLETeAh9Yb0fq5zBa2zRHDODl8nqsCCuOqsePOr5a4+C47A40PXfpjWvTpJt0047rB2ZU343Ao7gnXJMm6LZXVAHHsAsmgsmeGSDrEqjOxwasjN7D56t4H0mJzxPXt0u9+CxjfcjH7u9xR6r2h9K0cwg8kTKnW6Iw1Gi6g1YsEwV3v66K/A9ge//rx8BAPxPn3MfkkxqxQowhOcND5zD3kmMu0ejPDaflbnnRHE0ZgbPClnhahKziY7y54aBN0b9UOe5OmTFPOYVFDy643xkETyzSOGEtHgM9D20v9M6qV7aO9Hzi8uy4RkFzxTf864qOv+UfDmVRZNmFdlxl1I0Jyl4hTJzq+i8QsELfDVPutoZvKYhK81m8ABUBq1YRedMgXcK3tkEv9HUFqRuoe4wA1yK5mbAEbxTjsYzeDpFc4EK3pgUzUVst5ikKXOFhsgOT3GcBmYGTy3Qbtw7QSfw8OCFvrLGpVnlDN6/+LY34bvePttc4dj9YUXndEw7nQBSqnqDg0Ey9hy/5uoOvv2LHsG94xgPnO/hjQ9dAGCHqZAa+PoHzqnHbh9ZtQyAqZ2g8+IJuwevE/hWD545/7BmuIgMhr5XW3Q+SkiREhUzeEatKoas0PY4weMWTV7roB/PF+J0PfmeUhyn/eNE6h2dj2WA5i7jVFpJmrSr9gze9CErQEHBy0wiJ39OHXRSple2aPKfGdMfqZTew4EJnjg1KZqMoNUdNx1T1RyenRZrVNahI3hnEuYmVHsWvUnWvn12WD/aeDPDoYxGBE8IsSWECtoWQjwphPg6IcT8CSIOE2ErePXP0yEri1Tw8hm8cfNhs4CCGYoLwZM4RSYNoSQFad4evJv7A1zd7Wh106osYBbY1z9wDtfPVVcVzAMqOlel4/k5zY/xYJDgaDR+Bg8AvvcPP4mLWxE+/5ELePRyH4A9h3cwiCEE8Mgl9di945FWIEJm+wMMeYkCLycXRpXTfYqMfHHFJs1MZ17giYnzS1HgoRsqi6YsqCVRwNM57YUIt/zR35gsM48PJih4RXLSBL/xyds43w9LM4OLBB17kmaI2TETKQsLFs1pe/CA4gxeMWSl2TZo7pI/P2MEj34mw/yaWeQM3kdf2q+0b+t9nEXBqziPGSPWVQSPq8TcbeAUvLMJnRzfojUvOSLcQt1hGrii881AUzbwLgBdIcQDAH4BwLcD+LfL2ikHA65ejVOyltODp0jHTkWK5lzbrbFo0iKRLJqkOE0/g2fXJLyyP8S1na4OhBjEqUkLnJI8zgIiDHFq5uGIxB4Ok4kzeABwrh/iv/6VL8Pf/fqncWW7g63ItxS8g2GC7SjQ6ufRMNWL5D6dz5w8kIWuE/hIs8wifHT9RL5XCPxgPXipUfsm1SRQ0Xkmy2pJ5PsQVKgtpfV9loLHZstIlbEUvMIMHlUMTEvwnrt1hM++vote5C/NZshTXLmCp2fwtEVTKafTHIKxaJpvklK9f6Jgha2D7sErvC8AJ39mJjPMKz94iuY8CtcoyfANP/rr+Pe/83ztczhxGz+DN17B46SvitBz0sdJnUvRPJtoWtdympCkbqHuMD2MtdddOG1GUzYgpJTHAP4YgH8mpfyTAJ5e3m45EJrO4HWWYdGMlmPRNCEr9kLpaKg+pxTNWWsSaJFMi+lXDga4ttstKHiKbE2b0DkLeNE5hboQaT4YxHk1xORzfP/5Hs73Iwgh8OjlLasOgWyeROZORimO8/O5Fdnnk84LKXi0uFUKXl6TwGx4Hvs4y0NWJls07R48wCyiY6bu0X7RHxRakBxWWDQzKa2ZQYJW8HJCSxbCaRdiB4MEu72gZCldJEaM1OmahDTTf0gjnXjqzR6yUlCuSKRWqZyTLJo2iePb4uSvOIN3NKKbBt5cCtcgSTFMMuwP6rvGbAWv/n2apODx7VTO4LFr7NgpeGcedLm0ye6YVNz0cXCYBKfgbQYaEzwhxBcD+FYAP5N/bfFxgw4l2DN49c/rLEHBu7QVAQCu7nQWtk2Az+DZNixSbShFkytI00AViQttsXp1f4grOx1NNEjB81ch30G9J2kmMYhTbdEkpe1goGbwprXBPnppS4fHAGoGb6cb6AqKo1GiF6VEqOlmAS1mab9oIdsJfVOTEPAUTdg9eE0smlwVJGKdL4xNiqapwaAFOC1IiOBxQsKLvLmKQqSQCO2sFk0VdhNqS+ki8B/e87xV+cB78HhNgu5IJAWvomh8ErTNtcKiCaDR9uh7KaiGP7+YsAmon9GQJdGe74dzKVzFEKQq0HmLfG+ukBX+eNV1PEwybUXlBM+laJ5NtHHR68IyHGaBq0nYDDRlA98D4PsAvENK+WEhxOMA3rm83XIg8BmxRhbNBc7gPXSxj5/57i/D2z/r6sK2CRhlsLgQpLkrUrOIgM2ispFqdjRMcDBMLAVvEGdIU2mpo8vEbk+pdbcORyZkJT/Ge8cxhkk2dVLptd0uXsnL2wHgYBhjuxOg3zEBNtqiGZnycsDMFhkFz6gvdP1ErOhcWfzU62QZWTS9sRZNPv8X+UY55Y9pBU8IfYc5KaRoFtWoKgVPJ2+OaAYPU6tfgFJTd7pKwVvUIv7//NmP4id/5zP6c6NiSk1MVdG5etzUJDRT3AhSSjMYb6VookDwmlk0yeaaWedf/c9nMkNm5QWA871oLoWLlNM6ZRiAZWdtHrJSsR32GnUhK3Sz6WS0upRQh9MJ+llsk20t0TN4a94Rh1bBdKqudz8c5kOjVaWU8lcA/AoA5GErt6SU373MHXNQWGeKJgA8ff+5hW4PMISjbNHMCd6cNQmAqosYJiluHgwBANd2O+jQDF6yWgXv2q4Kbrlx90QfExG8l/YUSZs0g1fElZ0Ojkap6g3sqE698/1IETNP4HiUaNvcVqfaoqlm8AoWTasU3SioNCtHCl4n8IEknWjRJCsnYFQSbt+k/dI9eAWLph3yYRbtVTUJVormlAqelKquQhG8xSl4SSot8qB78LLMSrgjItVhpHcakmqdp4LqSZd5k1ROa87OE4VEzlzdY8X3YW7RJJzrhXORYzo/46yXnOCNncHj9tIqi6acQPDSDFsdH4fDpKDgOYJ3FqEtmi0ieKnrwXOYAVkLb2Y4lNE0RfMnhRC7QogtAB8C8BEhxP+23F1zAAzJARqmaC7Qorks1NUkkAKje/BmLDoHTDXBzVzlurrTRZf146WZ1IrWskHJnDfunZiQlfwYX95TXXbTzjmSbfbWoSKwNIMnhEAv8nE0TLXq0NeWVztFs0MKXmw+1z14zKLps/8zqRa+kZ7BmxyyQjcfSgSPBYpkBYsmzWMWEyEraxIqUjQ9MV0P3vEoT3DtBPnNgcUoeLwgG+AWTR6ykunjpPfIy+dDx/AcC3XJkkWL5qQ/2LaCZxPMpKDuAUoV5hbN3d58Fs0hq5GoA+8MHDdbNKnoPMnK74u1L7FR8Oh31XYncDN4ZxTattYihy4l9TqC5zAN2hgo5FBG0xXuU1LKfQDfAOC/AXgMKknTYcngKpM/hugEvoeHL/bx2OWtVezWXPA8URnGcLhABa8TeBgkGV6xFLzcorliBe96ruAdDhP9mltRACGMgjftDN6VnOCRQnkwTLCbk8StKMDJKNUkqV8MWWEWTZWiaVI1NcHzhA6EIbXGy62UwzidXHTOSBwpPNR/Nz5kJdPnCiikOGbji86LCt40f5z0tZcreItSadSMo9nWiJE6E7JiahIiS8Fr/ge2SITNx/z9m7w9TeJIQaxQwapSNGnf+3MmkBoFb7IyN5WCV5Wiyb42rJrBSzN9s4mOaacbuBm8MwptgW7RotfM4K15RxxahbSFNzMcymhK8MK89+4bAPxnKWUMwP3KWAH4nNikWbR3/fW345ve8vCyd2kh6Ed+uSZhUCB4c8zgRYGHYZzWKHipTtFcBS5vR5qk6uAYT2A7CuayaALAq0TwBrE+b/3Iz0NWbAWPFFFLwWMWQt6DF/p2yApgiNgwydAJPYSeqFRa3vPcHf0aSsGzw12q+vmqQlaklCU1qnoGL1fwCima01g0af5zpxsuTMGTUiVl8m0R2UtSqRWkhFk0taqpe/CavVadgielsWg2ClnRNsyyRTPl5I/N4PGKll7oL2QGbxxx4wpemkm8sj/AF/2DX8Tvv3JgPY/P4E1M0aybwevYboOdrlPwzirMDN6ad2QK0M+AS9F0mAZtvJnhUEZTgvcvATwHYAvAu4QQjwDYX9ZOORg0TdFsG/pRUDmDJwQLBZnHohmqkJWbB0N0Ag+7vcCqSVilghf4niZk/DV3uobgzWrRfPVgiDjNMIgzbHdUmEu/4+N4lJZm8EgR5ZZMawYvZAQv8PT595nFL81L0SPfR+h71iIaAD7xygH+xL/4TbzjfTcA2ApevUWThaykRsEZJlkpMIQet1I08yfR3yKPzQs2BSl4O51AB/TMC/ojWTWDF2cmZCXNTEBK5OdqqxB5D16zY0gLqp3+emYsmqKBgsdVOq9gES1WKABk0TTzv92wPqDmX/7KJ/GXf/K9Y19/qAlw8xm8z9w5xsv7A3zkJftP0qQUTf616hTNtGTR3OmGGCWZWzCfQegakhYtel2KpsMsoN9v7rJpNxoRPCnlD0spH5BS/hGp8GkAb1/yvjnATtFcFSFZBbqhh5PYrkk4HKa5ddF0bAGzWzSHSYpX9lUHnhDChKzEqZrBW+H5pKAVPve33Q30DN20Fs0L/Qi+J3DzYKCVTwpu6YcBjllNQpd12wGwLJmc4EW+r0N6Ak8wBY9Z/LiCF3gYpRKvHgzxr3/1U5BSYu9E9Ze9//l7+WuY4Ja4mKLJFLziDB6gVLWiRbOyB6+gImqL5lQKntrvbZ2iOb9KUyS0/OMkzfTjSZqxmgQz7yimCVmpCRTJpDQWTW+yIkg8p8oimkkJIZSiblI0hZ7B64YeulG9gvfBF/bwu8/dHfv6TSyaRsHzVapr/j13jkbW8/iMXZXViF9rdQqe7pWM7Z+xwYJCeBzagzaqGroHrz277HAK4G4MbAaahqycE0L8EyHEe/J//xhKzXNYMjipW0Up96pQp+CR2gSgRDCmQSfwcDhM8Mr+QKtd3cDUJKxSwQMYwbMUvFB/PK2C53kCl7cjvHowtObHAKPgHQ8T9CNfL/CDQl1B6Au7JoEpeFHgsRk89Zq+p54/SjNEvrJoJlmGn/29l/D3fuajeHl/UCoID31Pd7uNigoe6znUKZpsJXI0TEoWQT2Dx16naOejmoRpLJrcHkzq77yg168NWeEpmjl5ChgZW4xFc8qaBOrB81CyuaaZNGpuRU1CJ/DQDfxahWuYpBOJEVc466Bn8Hy75uPusV2OPilkpZFFMypaNNXPrJvDO3toY7Ig/x3j4NAUrgdvM9DUovlvABwA+FP5v30AP76snXIw4IRggwQ89CK/RPBU0XRgPQeYrdvvzY9cxIdu7OPDN/Y1uermCt4wSfMevNUljlLQCieV/FinncED1FzhqwdD7OfqEw9ZIYsmzd8BXMHL4HsCQV4UzS2bpibBQ1iwyPqe0AvbTpinaCYZ9nPV7mSUWsoXKWlawUtNyErgmTkuS8FjZOhwaCt4aqYtt2gyolBcvND82DR/nA6GRqHp5POb84IIhqXgWSErhsxkUub9cmSLbRaKQqialaOvmxm8ydsrztlZdk+uBuoUTWFZNOlntoogD5OsNHdbeo4uOh9n0TTXIL9+75YUvOYWzVGlRVP1Pfqe0PtNP2NuDu/sgX50pknnXSf4DHObSKnD+mEI3pp3xGEuNF3hPiGl/NtSyk/l//4OgMeXuWMOClaK5gYxvG6FSlIkePed6+HH/8wX4Kuevj719r/jSx7FdifAwTDB1V2l4HXWqOBRVQKPlN9hqt1WND3Bu7LTwc2DIVOflLrQi3wcDxOcjGxF1PTgpYrg5QocLW47ga/JdOgbG57vGQWIrGqdwEfge4hTqQnmMMms95SO1aRoGgWPd6dxpYiHthwOE3sGTDZT8KiIexYFb6cTohOqBNZ5QRZBawaPhYik7O56mqnzy6spmoSiEOxZRdtW6esZvOYhK7rovNCDR9syKZqebdHMf8aqCNAwzvK5yvqd4EXwdbAVPHPN3TkuEDxedD6jghfldSEncVHBcwTvrKFtqoYdULXGHXFoHehXp7sx0G40JXgnQogvo0+EEF8K4GQ5u+TAYffgbQ7B6wReaVFFpd0cb/+sq1p5mwbn+iH+9Bc/AkApXYCyKAaewCBOcTxKdOjKKnBNK3jmNXe04ubPRDavbHcsiybf3nGsFLweO3chS9HkJIgWq1zBiwJPX3uCWfxIyYgCD5EvEGcZ9k/U6w/iVNvluHJHBIDPpJEVVG3XKFB8UcItmlSGbmbwuIJnX0dEjsZwhBIoRXOr42ub4bx/3KrmBbWClxZDVqS2RQJULl9PTIr7VheykmX8/Zv8B9uasyvYXBM2t1pt0TQKXhUBovdsnP11mhm8KPCQSbPdewWCx22eVdvjNwaq9mmUZugEPiLfs1I0AUxUIs8i3vG+F/C2H3jnxi4K20bwJinYDg51cAreZqDpCvd/AfCjQojnhBDPAfgRAH9haXvloGHP4K1xRxYMCkHhOKwgePPgz731cXzuQ+fxlscu6K91Qx+DOMOdoxEubnUW9lqTcC1XEcOKGbxp5+8IV3c7uH000sEmtJ1eFOB4qEgsP580U0cL9SCfqaPFay/yNfEKPJ6imX+/J/RzO3k6ppTA3XxhPUwybeH8wscuYrenji8qzuClsqTgkaoTM7KmFLw8fMQXluWIz0AVQ1Y8MX3IyuEwRj9SqmSHpa3Og0qCpxWqzDrWUU66iUAFfvUM3vEowRu//xfwPz560/p6VrOYy4o1CRMOqThnV1TwtK2WBSFpghd6+mZMlYJH79k49csUnU9O0aRAILom7xzZM3hpau973XaqXo+Cb+hGR9GiWfzd5QA8e+sYn7lzvLFkgg6rJfzO7oFsy047nAq07WaGQzUarSyllB8A8LlCiN38830hxPcA+OAyd87BLjffJIum6qlTi6pbh0O8+9k7uHcc47PvWxzBu7gV4af/8pdaX1MdZynuHo/wuQ+eX9hrTcK4GbxZSe2VnQ7STOK528cAbAVvlCpl7XzfBLlY85yemvdKU4njkSpg7wSeXjSHfkWKpgdL7aPQFkov5Are3/+jb9DKmpnBMwpex1LwjFKUFi2a+R+Y0POQSqltd8MkhZQSoiJMhWL8p7JoMntwR/clZjOpx4REH68hA0T2ElaTACjyS8QUMFUPxT+w945jHAwSfOD5e/iKp67prxdJHSFOpX6fmoSs2HN2NklOpSzZdiPWg9cNTArrOAVvXNAK/U5oUnRORJwIXnkGb/YePJ30ml/neycq7ZZuWpyMWjKItULIDb/rLytcBqcZ/HepW6c7TAOTornmHXGYC1N51KSU+1JKKhv6a0vYH4cCeBDIZlk0zQzej//6s/iL/+69eHl/YM2lLet1ScG7sBUt9bU4rp0jm2h5Bm/aigTClW2lCv7H9zyPKzsdXOyr4yGb3K3Doe4UBGyCF3gCgW8UvH7oQwih92m7ExiywVSbk5greOrrhuCZeajru1285uoOgPIMXpzaFs1iTQLt59Ew0X9gAl9YNQmZNAv9YuKiJxQZnaav6mCQaAXU9CXOp9LQ/vEAD97zFhfSNT2vPINXXEzS97x4z3bI14WsJFmm3ydl+Szv5417J/jaH/5V3DwYIGPnv1x0zsk+V/DUx53Qm2DRVPs+zt6oCfAYfy0vOufbu3s8suyBkyxqSVZWVoufdwKVFkvfrmsS3AxeCZserU6XS1uOb9INDgeHOhi12l03bcY8Q0ibwzZOMXxrBm+NO7JgcIvmwSDBVuTj//gjr8Of/uJHl/u6oYdbh0PEqcTFrXDyNywIO51AWQAZYSfFaB6LJgC8uDfAn/uyx7RSQ4rgrcOhFd5SDOwJ8hm842GqF+ZP37+Ln/izb8EXPX6p1EPoeQIv3FXE4nw/0iSNuvyGSWopfASqSeAKHg+bsUNWMq2SHA4S/fXA95CxYBLALLL5DJ7Hetqm68FLNNHWCt68Fk2Wokl/KEdcwSvMgPmegE/dgEJUhqLQOXyhQPCsHjxObFharKiZwfu9F+7hwy/u41OvHiHN7GJ7e55Pgpy1JkXTWHm7gW9VkRRB53NcxYAmwGO8pDpkJX+to1Git8+toUlqyGrVopy/RDFFU3dDMqUaMLZql6JZBl0rbSFA08LUJKx5RxqiTtV3cJiEbMNv1pwVzEPw3Du/AnDVZZN68DqhCVkZxhm2uwG+821P4DVXt5f6ut3Ax8t7AwBY6QyeEAJf+NhFPHnNHB8tFmdJ0ASAK9tKFdztBvjWL3pEf51UuziV6LMUTVFIaaQUzeM41aRQCIG3PXklV5NMETmgFvzHoxSvu76Dtzx6UT++nweUUEpi5Hta4QF4yIpRtLiCJ1ggSppJhL7AdifA4TDVhCTyPUu1A6rVHt75Ns1d68Nhot8PUvDmVWmIpNB+SykLISuGVMRJZqVoenmKZZGQjRL1+Y279QoeJ2VxljGiXm3RpP44leaZWVUIJYtmIUUz8oV+L7mCV6XSUQXCWIsmI8B1oH3q5MTreGi2x8vO49RYgatG+pooeJHvVaruTsErY9ODGTL2O6oN4D9DTolxmAbphv8snxWMXVkKIQ5QTeQEgN5S9sjBgqW6bBLByy2aUkoMknSuWaepXjf08MzNQwBYqYIHAD/+Z95ifa7tkHMoeNudAH/+rY9b9RK8+65fII9UVu7nM3iZVFbIXsX5L6Zo0vX3177iSXiesFQ4QKkvw9ierwPMDJ5l0eQhK4xIxLnitNWxUzQDX1gzeABX8BjB4916U9YkkOV10QoekB974OXF44p4xaUZPPPz7gtRSchIwXt5f6ACUfLnF7vv+D7wWUp62nO3jvDfP/IK/vzbHtchOXGalebsLOLIQla0RZPVJHQCE7JSReK0gjfOohmXSXsRZgYvD1lhZOvecYwH80wlCkk5GqVT1yRwBS+qUPAcwSuDft7aQoCmhZkxbMfx8Z+htnT3OZwOuJCVzcDYlaWUcmdVO+JQDU7qNmsGzyQrDuK0RAqWhW7g60j8C/3VzeBVYd4ZvG7o411//e240LeJ6habu+MzeIBShIfIFbx8YX4wiEvPo+cC5hrc7QX4nAfP6XAPnoQJKOvdIElLxfRCKDIY64CU+h68NFecOmGAwyGzaOaErVLBY1/jCtO0Ch4RbboWFzWDB+Q2zfzjrUj1MxZn8EhVBYyCV1wrk+qUZhKv7A9w/3l1n83qwcvISiatmUZeu/D/vf8Gfuh//D7+1Jsfwp6l4NkqqJQwYTac/FWkaHZDX98oKCp4SZrp8zFewZts0dRF53oGL9GPWQpeJnOyHleSfetaKlk0yWrs1yh4bsVcxKbP7bRNoeQ/Q26h7jAN9O9Ld9m0GstNtHCYG3yht0H8ji2ila1vVQoe7767uMKQlSrMO4MHVB9Dj5G1ov1TzROluYJHBC/B1Tzlk8NnVj0A+JFv+XyEvqcVvTLBIwWv/F6GvmcVnfMwHW6njHNVaqejCB6tS0LfQybtGbyqxEUeDDPNXev9QazfD61CzbmI52rjKDUErxf5OBgm1vZHiZ2i6XtUTF5t0QRUOAoRvKqQFT6/CBjCBkB3F94+GjIFTxbm7Gh+TamsSaFCAVDqrOnBMxUTxaJ4roaOS6BsYtEszeAxi+Zd1oWXZsYKXEX2aREjhLLIcvCQFbIiB54wBNYpeCW0jQBNi7bNGLoZPIdZQX+63HXTbqyu6dlhZhRnoTYBmuDFSsHrVpCCZYATyXUTPD2Dt8Duv+L2ehUKHv1PH++fxOhXEGxauNNi/tpu1zpnZYtmhmGFgkfbIsWqZNFkdsokzRB6HrY6gbJo6h48ZSfltsaBVnv4DF6+TdHcoimlxOEw0R1ni1LwihZAIg30/nCSMEwzeB77WRfVPXhc9eNJmlWLuUQTPDZTlz+2P1Cq3e2jkT2Dx+fsPHvbSWqUV3os8I2FsRMYBW9YIECc4I3vwZtck0DvK73ucZzqOhBelZCkks3g1St4vdAvhayMmEWTrnMqdQ884SyaFaBra1Mtmm2zrVk9eE5wdpgCm36z5qzAEbwWgIdcbArMnFOKYZJVkoLlvK56HQryWCcub0f48299DF/J+swWAT5Pt9UpEDwWuEHX1f4gqbRoTrruqhS8QY2CFwUeRixVsq4mIc0kAl9gK1fw6OuBDmphlQNawcusbdH/pNrcOhziH//Cxy1FjeN4lEJKo6TqovM5FTxe3zBkBK9fUSVARed+yaJZPYMHQCeaAvbzSgTPmsFTXzsggnc4xL1c9UqyrHLOTm8vlQgD+5oIfGP17fKi81GR4JnPx/fgUY1EcwXvZJTg6k4HQgB3jmPrefRejpvB60d+7QyepeD55ry42Pky6HxuqkWTDqsti17+M9QWUupwOtC2mxkO1XAErwUICizjQTwAACAASURBVFa5TQAtvEZJVksKlgFagF7oR2tPJRVC4G987VO6L25R2KoJXAFgLVbpujocJlbapnmuPW9VRJHgkYLXrSDrUVHBKxad539H4jwUZKdA8EKvrKqZTjmpX5MTJFJ6/tN7X8A//aVn8JGX9lGFw6GyK2538hTNoD4oZBpw4qluZKjtkW2Wk6BRkto9eEKoYxhD8G7UKHj0FCK0pibBhKyQRfPW4Qj3clKUpNKaszMWzZyYMwWPHrMtmj5C34PvidK542S5UQ/e2Bk8CU8Y6+nRMEUv9HG+FxYUPKMUV3F7OmfdsEzwuIJHxI62NY06fJagUyY3dFFoahLacXxuBs9hVpibNWveEYe54AheC+D79h31TYA9g1dt61sGiOCt2565TPTHzODxkAyfdfIViSDAZrdq3pqiRXMQ52psRWAOD1kZVYSsaItmliHwuUUz//7ArlpQr2fmtYicmcRPs9D8wAt7AMrl4ARSsxau4BVSNIfaolme44pTqVRV3yiQVcXkpIL2I986nmKdAWCUrpBZNGlxejAkBa9g0czKVQi8o9BYNI3VVxO8/Lz1Qr80Z8ctmuPSSZsUnavgGE+T4eNRgk7g40I/smbw4lTqGwlVhMxS8IoWzZRbNEn1N8fuUgnLMN1Za96RJaFtFlQ3g+cwK9o2b+pQDUfwWoCgcEd9ExBxghdnK5vBI/KxyQSvE3ha7S3N4DECwTsWx6Vo1l13vAC6F/oYJBmGcTo5ZKVKwdNzXkpBUjN70lg0PaP4RoUZuTST2M5JEw8BoW1+4Pl7AIAb9waVx0Gpqjt6Bm+6moQPvnAPn/t3fkEXvhNKM3gpWTTLXWpk0bRTNEVJLaAwkEcubVldeMU6A8CQJCLyXqWCZyyacZYhk1UWzfy1U6nJIhG8MPAQBVSToM5bN/RKCh4/1vEzeOW5yiLSTMLzzD4cj9QNogtbUSFkRSLK96lKVdIzeFFQYdE0KZp0zFWzjA4G2tbVEgI0Lcyid7370RRuBm91OBomc89snyY4i+ZmwBG8FqCYZrgJ0IvoOF2pgkfdWRc2mOAJIbRyV5rBYzNq/gSCN2kGjwelXN3t5O9lnYLnWQqeHbJiz42FvkAYCIzSTC8WaZE9SjI9O8ntfGRL5QplmkncPhzqWbU6BY8smjs6RXO6ovPfu7GHvZMYL+/ZBDIupGiOm8Eb5hZNroxV1STQNh+91MeNeyeaAFbdrafnjgtZ+cydY70QJAWvaAnPKhQ8uiZCzyspeN3QL3Xd2SmaTXrwxls0uYJH19yFfog7R3wGLxsbskIW2l7o1YashL7QNzLomg18b6yFlOPnP/wyfvGjrzR6btuRbviisM09eG3Z57bi237st/GPfv7j696NhSHdcDX+rMARvBaA1ItNtWgOVqjg0eL94po78JYNUu5KNQkskZX3e/UqLJpFtab0eGDIyPleqBS8msoLK2QlLYesaAUvX7xHvockzfQfGCIRwzTTpHXIis6J4JGdlLb5wdyeKQQsxYvjMFfwaBu0kG+q4N06MEXhHEUFz1g0yymacarqCQKmto0rOr//fA/Ho1Rvkz+vWJNA7yPN4EkptWr5zM1Dax94D54uUdeEUVakaApc2+1CCOD+c6qyoRv65Rm8piErDRU83xPW78Mo8HBxK8KrB0ZFTTKTojkuZKVXNYOXmiCXkNRKRm6bWjT/r3d9Cv/qXZ9q9uSWo20hJNNCK3gtOUA3g7c63Nwf4ubBcPITW4K2zZs6VMMRvBZgs1M085qEVc3gBZuv4AGGRNRZNAPPs4jbVqWCR+Ec1a9BZGSnG6AT+hjGaW1pfeR72l5YVPB4D16SZnmFQ16LkJEKZSyaRFp5Zxqpb9yimUmJ9z9/D54A3vTwBby4V03wjnJFiZTBYMoo/FcPB3o/OPgddJ6iWTWDV+7BExCsaFw/L98mlduT+sgJh1FDs3xbREzUto5HqSY3dlCLsmjWhazwegs6z6Hv4clrO3jv3/wKfNZ1FRakZvDGhayMmcFjc5V1SDK6RsyF2Ql8PHltB7cOh7idW2UTNoM3riahX2HRJAUx8j197Rl7c/NFfpLJkjq4qWjbjNq0aFt0vK3qr3FHzgCKHa1tx6bP054VOILXAmykRTMndMejJL/TvqIZvPx1L204waOqhLqQFc9rPoNXl6JJi/3dXpirNllt5QVZLtNMIpOoD1lJVU0CqYO04CcVZZSkmrzqzrSUK3i2RfMDL9zDa65u47XXdmoVPCJyfL+7oT+9glckCTU9eGYGz7ZwemwGz/eEPu/8JiopeOdzBfpIE7xyiiaFvISMsElpZg6jAhGPU4mE1yQQwdPbyyw1EDDqIL9h0g29Ukm81YM3VsEzPXh1d49JwfMtgufhqft2AUCnpcapSeettmjSDF45ZIXOs+q+sxU8XzSvSZBSlsjjpmLTaxLaFjxhzeC1ZJ/bikzKjUqc3HS79VmBI3gtwGYqeOrS288Xm07BWyy2Oj6EKJ9XqhsIPDtFs8qiGUxIb6XHd7shuoGnZvBqSutpBo/HzxP4glmpM16pFoEW16M0Qy9Ux0bELM4ynYDpM3vhwSDBb33qNj73wfN44HwXt49GlaocfY33B3YCr/HQPIWrFEkCr0kY5WmxgFFLB3FqkWxOWpRFU32d/5ElEnkhJ3hE1iyLZkFJ4dvMpNTzd49c7Bf2VyLLJMi5q4vOScFLMq1m6VlBv/xz2w19S50EzPu43QlKJeiELFe76JzUqXhE8AJ2/UaBh6fuzwnei/v6eWMtmrLeoqnJsW968OiGBq/gmIRUysY3CtqOTbdotm0Gz77p0459bivSrD3XRRO0zY7sUA1H8FqATU7R3D9Ri80qW98yoGsSNnwGrx8F6Id+qeuPL/Y5uai2aI6/7oh0aYtmkmFQp+DlKZojrYyYbaoFs/o4yYvO6XFS8AIWshL4IidgRu0heyXt6+XtCEkm8SVPXMZ3/6HX4oELaj6sKmiF7IR8drAb+iUVqg63mCWQg9ckDHnISsekaPLXVN1uxgZYTLFU28wgBHCupyyaVQqeZJZKwLxPIg9ZoZ+5xy5vWfubUE1CScHLt5dJFGsSilUZAJ276pCVc72wRP4IdG2QGlu3KE20gme+1gk8nO9HeOB8Dx9+cR9SSmsGr7IHj1VO1PXghb6pgSBV2Z+i6DzNcOYUvE0lE21LCeUzwRvEPU4l5IZaNN11026Ub9s7nDpspEUzV3losVkVzLEMPHq5j17o4/ErW5Of3GL0I18TCQ4zg2db3IqzeoBR++quO23RzBW8w2GSqybVIStcwesUFTxu0fQ8hEGNgpcohaebz/wBasHfDTwreOM73/YEvvktD+PSdgeAIXY37p3g8Svb1r4NktTqc6P9a6q8ULDHpJAVmZ97ss3GqcS5ngdqV+Ck2xMCQpQVg1EedEJzfEejCgWvsNA2KZqKmJDqxwmeEGruLJVSE7vqGTz7sbCiJLHK3kqE73w/rCXOROa3Ih97JzHitDqwh5I+uQJN19xn37eLj7y0r499nIJHCmE39JHk6iVdP2RHFcLcbNABRex6nYSzZNHc9Gh1ugnVlnW868FbHVIpW3NdNIFJ0dyggzqDcASvBeBzOZsCY9HMFbwVWTSfvv8cPvp3v3olr7VOvPGh85Vf19eSX5zBK/8q8CekaBJx2OkG6IY+I+s1ISuptAqkCZ6HgkVTMItmZu33KMnge0rBG8SqRkFKFSSiSB709oncASp1EqhT8DLLnknfX2cltL831SEtceEvfFywaEqpjqHf4VZQ87HPSIvvCXhZ9Qxe5Hu6s+9waJJECaaYvGDR9IAkMRZNInjbnQBJlmmLJr03OkWTtscK6rc7vibaRfRCrxyywhS8vZO49D3qOWl+fpopeHbIitqvp+/fxS997JXSnGFd0bknzO+eUZqh66nj4ZUQQWEGjwJ8miA9gyErm7ombBuBTSp+JzgsB1kmW3NdNEHbOh8dquEIXgtAC62i3a7N0AQvL1xeVU3CWcFf+ANPVH7dZyrEpBTNYMJ1F7KQFQGziK8uOhcYJZmeIeNqmSfGhKzQ8wOzCA98D53AxzBJ9SIm8BXZqAuEuX6uC09Ul52fxCm6heOn0JhJ4OXmxZCV1ErRTJFJdQw8+IaT4WLISuUMXq4skY2xyqJZTNHk8f6ZNHOvRPDO90PcO45VTQJP0awqOs/fhz/2+Q/i9Q+cq1R+K2sSYkPwXt6vLpwv1kjEafXqIs1VXm4dJiL31P27yCTwoRf3rK9XWSpTaSo5gJzghUTwjB1Vz+CRRXMKBS89Uwqe+r+pfbVtMDOG7Tg+27a9xh05A8hke66LJmjbzQyHargZvBZALWbWvReLReArSx2pCauyaJ51hMyiaffgVRE8OxK/alsXtyI8cqlvvX/jis6rFDw+05Tk9jt6bVLRQm2dk7lypCyUPEikG/q1gTCh7+HabrcySXMQpyUFr9NQwXv1kPeuFYI62L7xHjyeWGrN4JVCVmyLJGCUpSLB48+hdZ0mv4yoV83gne+HCHyhahIybtGE3raUSomi92GrE+DzHr5QeU4qaxKSFKEv0I8CqzKh+BzA3GyoI1GpVFZKfv3SNUdJmtR/GPqemj2sUfB8T+hrkROxEVPwihZNz5vGoonGYT11OBwm+Oofehc+dGNvru0sG5tu62pbTULClONNfU9OCzatJsEk4q55RxzmgiN4LYC6m79hDA/Ktre34pCVsw6uzvAZpiqLZugLfMMb78cXPn6xcltCCLzzf/1yfMtbHrbev/qi80wv7os9eEbBUwodLa5JReNpjcqiqYI8iFQR6asjo4Cyab5U0YVX1cPYaViTwIu1RwXFKc3tpp3A0zUJETs2wD5XvgCu7HTwNa+/jjc/ekErp3zdMEryGbz8/SIrIncB8nlGgJenq20dDBJEgYcrOx0EnsCFfoTAE4gzW8Gjc5lmZvHC970OdO44qRomqrKgF3m1ISs0m2cUvOrzb2bwGMHLz+ODF3qIAg+fevUoP3ZRW2uQpDnB88sEL07MvGGgiZ5JoG2s4GUScSrnCuZ4ZX+Aj718oOsfTivaFkIyLdqmalgWzZbsc1uRZptVk9C2a92hGs6i2QIEvqhVJtqMTsgInlPwVgK+SCVlpxN4lXN2Qgj80Dd93tjtUZrjJAUvyhU8CgXZZgEwvieMDTBP0Yx0D54dskL7TiEohsSMV/AAVQ7+0p6yB2aZSlmMAkU4igpeN/Bws4GCN86iqQlETm4zqYguP9ecWPp50Ms//7Y3AQB+85O3Adi9YnGa6W30Qt9YNLmCx8gyYIesyLwmYbcbQgiBKzsdXNyKEHge0lQizVBh0ZSmNqDBjRg6l8Mk08rwMEnRCTx0g3LCJkFbNCcoeFUzeETShBC4vBXh5f2T/Ni9XHErb4dK3asUvDjN9LHStUfXpOcJ1LhHS6Bj4PN904Lez9Nu9WybwjUt2hYd74rOVwcpN2vO0RG8zYCTTVoAPo+zSegE3sprEs46bAUvD/2osGdOC05U6moS4lRqQtIvEDye/GhZNFlcPSHwPZ3USHep/Tx4ZJzVd7cbakvwj/3as/iqH3oXABWUUvy+TkU3WhWo5BwoWzSTvFYgyisihkmKKPCsY+HEsjjr6FUoeElmysa3OoFJ0ay4W1+0aHqsJmE3D2n5gT/xOfiut78GvicQ5xZNXxhCqLYNVm8x+eeUrgVO5Aaxmm+jCoWqMmxt0cyvjfoePKWMclcDv+Yu73Twck7kScGrTtFU24nYfCeBz+CVi86bL/LpOOcJWtE9hKc8rMWkTG7molC2jMDyn59NLZ8/LVApmptzjo3des074jAXnILXAhQXM5uCTuDj1qFaILsZvNWAFvtcwauyZ04LHqxSV3SeZibBcZslSVJ8P3UJBZ6xMRZTNOnjTuDh7vHIVAF4At//9a8feyNkpxvoUJ9nbh7i2VtHyDKJQZzifKEXsWlNwq3DIbYiH0ejtBQKkqSZVoiGeb1D5HuW3bRjWTSLBE/9b9ctGOKx3fErUzS1gpfZ9lYhVN/gwSDBTq68vvW1VwAoEpMWLZqs6Nx06k3+PUSk9SROQVN6yqLpoRf5yKQiPMUwnmLISpEwE5K0agbPbOvydgefvHmYH7u6OVZ1dz3NaxGqLJp8Bo9uNgQs4XSakBUgD5npNvqWEug0tEbB29BVoUkWbMfxuaLz1SHbMILHLxcp5UYF/J0lONmkBfDzu9Cbhk7glfqqHJaLgFUfLFLB60xS8HJ7290jRfC2LAXPWCYB5J10lKKZ5vtdmMHLQ1aIBPiewJPXdvCaqzu1+7jbC3EwiJFlUluDj+MUg7hck9ANvVorIcerB0NcP6dW7kWFRSl4uUUzL3nvhJ4OKgFsMly0ydIf1aJFUxO8bjA+ZIXZVwGm4A2Mgsdfu9gFx0NeaFvNFDx1TPz8DWOlXtLPeVUXnunBC6z9LyKTFTN47PfHpa1IV1cEnlcbikJqcbWCx2fwcgUvMOdlmqLz4ranBb23rSF4m7POtdA2AhtbIStr3JENh5Rq/m6TzjG/xjfpuM4a3Kq6BQg8lQS3aeBEwCl4qwFPxqSPF2PRrO52I5BKcu+kguDlC2ZNSHjISlxWjkJfzXINk9RS8CZhtxsik6ocnAje0TBRM3iFc9AZMyvGcetwiKs7XWVxLBK8fAZPVTqYkBVbwTMfF3/Gqyya1IMHKCJ0WFGTULJo5udut6tmEG8fjrDbDa3XCjxPF53TqTYWTa7gNbdo8jCVYZKhE/r6PFedW2PR9PX+/8fffQHv/PhN63k0g8dvevFU1ss7pv+QiGBd0fnYGbyCRZPOu++J6S2ac5Az02t42gme+n9TAz3a1g1m1yS0ZKdbCH3dt+XCaABrpttdO62FI3gtgO9tZsgKT1KsKsd2WDx40TkVmS/GoulVfkygRfS9Y2XJ5V1wnicgJZ/zEsyiWQ5ZIQVvELMZvCYEr6dec3+QaKJ5mBO84g2GXt6DN2lhdOtwiMs7HYS+KClOSW431QmiSVYOWRmj4NX24OVK0nYnwOGgQchKTuS/7o33Y+8kxmfuHOtzQQhyiybZH/n+8LLuJhZNo+AZQsJDVtRjVQTPtmimWYZ//svP4P/5zU9bzyvOaQL2NXeZFdwHvlfbW0fb6VQRvKTcg2dZNJsqeIsgeGTzPOUEb9NrEswMXjuOzxWdrwamUmBzzjG/XNpyvTuU4VbVLUDgbapFc7zq47B4EKnzBZ/BW6yCV6XG0mL57nGMXuhbZIaubVoE+9yiGZdrEgJSxWKj4DVRlki1OhjEOtznaJhgMCrXJPQiX0fcj8PtoxEubUUIPa9kw0uyDIEv0PE9jJIU+ycxdrqBRZKiwKjz5Rm8cg/eiIV/VIWsRL6xPSf63KjtfOkTl/HQxR4AYKek4AkV5y8rQlakUY+iKSyaw6KCF3iV5I8/BzDXY5xKDJOsVKuglFFPX8tAcQbPzFNSAnGdgqdm8NT3lmbwAlvB4xbNpgqeTtGcg+C1JUWTFribtNDlaJsFNc2k/t3Sln1uI+i62CTlOrPU3zXuiMNccASvBfA9sZFDrmRP80QzZcBhfvCQFT2D15lfwWtSdA4Ad49G2oJHIMWICBK3aNKiP2LXB5/BI+LRTMFTpGb/xFg0D4cJBkm5JkEHhYzqbZpSShwOE0XaAq9yBo9mvEZJhlcOBri227WUp9AXCD0TgsJBn/I/sHGS6e/f6pgZPHrp0Gel8amtbnqewDe++SF1LrpFBc/TfXembF09xmfwggYEj4esEChFsxeV7ZsEIoTbWsFTBO+48B6oegPblsutrle2CxbNGgUvy98fnfrJCsn5DJ6uSfCnV/DoafOUnS+CJK4CdE5OudA4M9oWssJtxm3Z5zbCzGaueUcWCLtiw107bYUjeC3Axe0IF7fCyU9sGYgIdAJ/IwnsaYSxmXlGwVvA/KNt0axS8PKQleORNX+n9sXuvAtYQqIOWbFIkafn2opBIuOwk5OaO0dDPbu2fxIjTmWZ4EVlklLE8SiFlGq7gVdh0UwzbdEcJhlu7g9xNbdzEgLPWDaL3KlKwVM9eGTR9M0MnjRkjp4eZ+W5uT/55oew0wnwyKUt67VohjCT67dokmU4TjMM4rT0XLK+8mRhrixesgiel6delvdTzeB5+r3mRJIvjgNt1TTnpS4ApoiFKHj0fp5y5rTpNQltC1lJM4kOEbyW7HMbsYnW5KrQLof2wdUktAB/9Q+9Ft/51sfXvRsLBxEBN3+3OgSMTNACvhgwMgssBa/i/ezoGbxYl6MTyBLIKxGigoIXFBW8fHu0KG+k4OW2xBfunuivvVpT09HXi/6kdntErrY7IUK/bNFM89L2TuDh1uEQwyTD1Z0uhBA6aj/whUUcOIjTFkNWTE1CqOYQ07y/zjPbBYC0oOABwLXdLt79N/9wSWUNfYFhnCkFTxjFT72+1CXuTSyaVQreMFa1CN2xISu2RbNOwSOVsU7BK1s0qxdfajuGUJ5YBI/14Hlk0TQBRU0Xc4uYn2tdiuaGrgilVvDWux9NkWRSzT4P27PPbUTblN0msAne5hzXWcNSV9ZCiK8WQnxcCPGMEOJ/H/O8Py6EkEKINy9zf9qKfhRYd6U3BRFT8BxWAz2D53nM6rfgovMxFs17xyNtwSNoi2ZiFCdt0YzLKpSy1dkEjCt8dSCL5vN3jvXXbh8O1f4XSK7Zfr2Cd0CdfvlcXVHViVNj0XxlX73O1V31c0ykK2RhN/VF51zB4zN4ah+PRmmefiks+2BckzDaDcuKue95iEsWTUbwqCahQZ1JVdG5StGcpOAplY/IfJyq5NHSDB4VnXtC21j575AL/UgfwziLZporeP0KBW+UlBW8kFs015CiOU/VwirQthm1aWGOrx0HmKZS/yy1ZZ/bCLqhsUlBNnYC6xp3xGEuLI3gCSF8AD8K4GsAPAXgm4UQT1U8bwfAXwXw28vaF4fTCSICTsFbHUJt0VTkYqcb4Pq53tzbpQW2Cg0pq2m0OD4apSVCSeLckIWs0MJkkBjbpn4+U/BIRWui4JFF83mm4N3KCV7RotkfozQRDvIEy51OgNAvz+Aphc6zVK+rO6ozL9QExFhliyErVT14I0vBU8dzOEzy/joqMzcpmkHD+d3AE0izDJk0xI72J80wXU1CxbnTFk1N/qp78DqBIfdHeYl7cQ4yTaVF4AD7poLnCVzcivT+ejUzczxFUwjghKm1Sim1e/DofVShLRNPg34NYDEpmqdfwaP/N3NFuG6C95EX9/GXf/K9ja26xsq8ue/JaQCd2006xcWic4d2Ypkr67cAeEZK+Skp5QjAvwfw9RXP+7sA/iGAwRL3xeEUwlg0nYK3Kph5L9XH9ot/7Q/gm77gobm3Swv3bo3Cw4lBcQavrOCJsoLHthv4XsmiGTSYDQt9pdZwBe/WgbJo1oWsjFPwtEWzGyDwvVLiZpxmVs8aYBQ8PdflC/D4fQ4TcmJvk8I/6DweDRNtreT2Qa7GTQLNEJJtFOAWUdOD12TWsUqlUyma/tjZRurKo32mhNCTOLUWGaksq4xF6+ilnOBRX17VIiXJMvhCEeB+6NfO4NFNETovvmh2t15Kqd+74SJSNBvO/a0LtJ+bSibWPWP4j37h4/iZD76El/eaLZUoxdebwlLsMD10uNAGnWM3g7cZWCbBewDA8+zzF/KvaQghPh/AQ1LKn1nifjicUtDcTJWlz2E5CNkiFQCu7nYbqTKTQGS9U0PWKRgEQNmiqWfwTKBKqQePWTC5RZNSJJsQD0CpeHwG7/ZRbtGsqEkAxoesUAfddidA5JeLztNMIvRtgndtN1fwfKPgFckKodKimVQreGkejsIDRbidcxICXyDJpNpOsSaB1UVEDX5WQ1/AE+bcSSkxSjJ0J1k0Y6Xy0XtNCh4PeaHPuYIX5Sodx5W87Dz0Ra2lkm+nFwU4jqtn8B662MPrru/gddd31XlpaNHk672FWDQnJHH+t997CX/u/37PzK8zL9atcC0b5sbJ6l/7mZuH+KWP3QTQPGwnya9vRfCWuXdnG2Y2c3NOskvR3AysLWRFCOEB+CcAvqPBc78TwHcCwMMPP7zcHXNYGXSKplPwVgaf1KIFkDp7u6q7ro6sU9cYUC5W9wsKnp8vypW1yH4OfVy0aDaZwQNU0ArNw4W+wK3D8QreuJqEg6EheFUWzTiT6HueJr/9yNekLGCqkFbMChxVK3hss3EqtZrJFTwKWeGBIml+B78JAk/VJGR58AjAUjRZTUITwiiEUEXxufpK6lUn8NHv+PAE8Onbx6XvOxol2IoCPZNI5B0ABqNMn0eqn6B9LBJjwJSd+3naZtWaOM0kumGeJBv51ns9YoXy5/sRfu573qYfaxqywu/oTyJn40A/A5NI4u9++i5++eM3Z36debH5NQlkxVv9gvfHfu1Z/XHSkK2lqUToeep3gmN4S4NO0dyg694VnW8Glimd3ADAvV8P5l8j7AB4PYBfFkI8B+CLAPznqqAVKeW/klK+WUr55itXrixxlx1WCa36OAVvZeA9eItGN/Br7bahpeDZz6F9IcWHFBweasH5W+AJfVNgmhRNwAStAMD1c13cOqgOWdHJig0UvJ1uoBSwgoUuzTKEzKJJ6h3fX1KYqo5BFBQ8KWXlDN5RruCRRZMWHDEjQpMQ5DUJtB3ALjqfpiYBULZrOndks+0Eiux+3efej5969/N6/pFwPErR7/h6nw8ZwTuOzcdpXnQOoGSBJVCSJvU91qdoGoJHgT0yt6TWJYY2DVnhz5knIEWnaE7YRpJJJJlc28zMui2My8Y6Zwx/+v039DXdVA22FbzNfE9OAzZRuXZF55uBZa6s3w3gtUKIx4QQEYBvAvCf6UEp5Z6U8rKU8lEp5aMAfgvA10kp1+cxcVgpIh2y4hS8VcHMES2e4HVCv5asj5vBdQERXwAAIABJREFUI6JC5eOG/OShFsLe38D39KyfVvAaEg8q+O6FPi70I63CdYM5ZvByBa+4AE/yMBA6J2QbVMdWDlkp2gw9HbKSby//o0szeMaimSLN1Pd7ghedZ42VzSC3mEpp9sMoiEbBa1KTAKifabJhks2WLNl/5Q+9FsMkxb9616es7zkaKgWPzgdX8Li6llgqo1d5zZGCp0NWKghZwiohepGZwUszCSnr1UqvhjAWsWiLZpyMf01SkNeV5rdOhWsVkHohv/rXPR6leOBCH8A0Fk0+g7fMPTzb2MQbG9x9sEnHddawNIInpUwAfBeAnwfwUQD/QUr5YSHE9wshvm5Zr+vQHpiic6fgrQp1atEioBSaOotmPcGjz+8dK4JH5Cf0jZLEyY+l4E2RogkYBe9cL8QWs4oWuwD1DN6YHryDQYxe6CPIax2SgkcnydQcFx37VUbwTPS+CVmptWhS7UEhyZLSSA8HsbJW5ueJFqIJC0yZBN/zjEVW2NdImrGQlcYEz9PKHbdoAsATV7bx9W98AD/xm89Zs3jHoxT9yNfn44ide060VciKKR+vsnj/gc+6gj/yhuu4tB3VpggWFTwikfEEO2pd7UJp++w15wpZaargpeZ9XwfWOaO2CmgFb8Xnl641uqlVDHMa931kdd+kCP/Thk287l3IymZgqTN4UsqfBfCzha/9rZrnfvky98Xh9KHjFLyVg9seF41uaObNiuA2umLIynbXVvBMwqTZV67gcVXsiFI0mxK8vOz8fD+0iGZxBi8KlLI21qI5TPS+h74oKSxJIUWTWzSNVdarVVWLISu0fUPwcosm9eAVOt+SdDqLJpEQo+DxHrzpLJq9iFk0ScFj18CbHrmAd7zvBvZOYqtzcKsT6PNBISuAHcpC9QZAbtGsIGKvu76Lf/atb1LPGdODR6/VC33cPVLX3yQ7qu+JRiTKsmguJGRl/Dbi/AZDnGZr+Z2abqCSwbEuKx5da/SeJk0VPD2DV50i67AYpBuoXPObGG5+s71w0onD2kB33l0P3uqwTAVvtxfqrrkimlg07x2rwBNavOveMWEHaVSlaDY9Htq/3V5ozQIWCR59bXzReYIdCk3xPb3AJpCCRsTmqmXRNCErOkWzaNH0zAwcwIgHU74DT1g9eB5L0VQWreYWzRHrIbRfXzLS01DBC4xFk8JWOOnoR+UQm+NRgn7EahKGZQVPSruM3feEtn7Woc6imWYmMbQXBZqQEpmtSwz1PdFo0SMXpOA1LTonBW9dao3cwIUuh6mBWPHrSiJ46npsOs9JPye+s2guFXS9b1JNAj+WDTqsM4e1pWg6OBiLplPwVoVgiQTvB/7459QuirkaslWwQ2qCpxU8u2DaE7BDVlgPHil4TYmHZdFkRLOKJPQif2zROVfwoooUTVLQ6JxQBx5gzr81g1dS8NT/smDRpBk8IQS2OgGOhomeJ/NZiua0Ch4pBdqiyYvOC+rhJPQiX88okoLHrw0ieJxAHw2VRZNeg4esnMQp9o5j/NNf+n1rW1RUPg6+ECX7LGCncaoePPV6k0rdfa+6OL2IRYWs0EtNUvASreCt26K5mStCeh9WvZBPCwpe0/c3zjJ0wgBCNLteHWaDVq436LrPMhMmtamK/FmAI3gOa4OxaDoFb1UIlmjRfO21ndrHxil49PndfAaPFt06ZMUT1v7yRf3UM3hdQ/CIWApRPQfKgzeqcDhIWO1BhUUzkwh8z1g0d4xFM2Qktk5V5SmWACqrCrYiH0fDVPfgedyiOcUMHlf6tEWTFZ0nWabCbhqe507g6woKqqWgFEBAKWYAcJKnY6aZxEmcoh8FmtjyGbyTUYof/B+fwE/85nP4o5/3AP7kmx4EQHbd8TeIfE9gWBFQohQOL98f815PqoTwhGgUiW7XJMyh4NEM3iSLpp7BW88wULomhWtVWFeIjJnBm86iyWfwNlVVPQ0w1t0178gCkeWWf0fw2g23snZYGyKn4K0cy6xJGIdxISs6RVNbNAs1CQWLpl9h0Ww8g9dTr8UVvF7o60oCjl6ogjde2R/g+/7TB0tq3uHQELwwqApZyRB4Ap99364qyr5vVz9mjlGwtNBiTYL6vxiywYlHJ/QxSjMTssIi0eNpUjR5z2D+IZ/B4/UMTdANPX2+nr11BAB49NKWftz0DKpjInvkVke9F6EvrBm8kzjFC3dP8FnXd/GD3/hGXM3nGetqEjg8T6BK9OCzfDxkZfIMXjMVhz9lNRbNLP9/XRZN9f+mLghNyMpqX5fefwp+aqoGKwXfyy3FS9u9Mw9zY2Nzrvs0kwgLIwIO7YMjeA5rAxE7p+CtDnWR/MuG5wn92sWQFd9TxdhFi6ZO0SyUWfNC9cMZFbzzjODVBVJQUMi7PvEq/t/feR4furFnPX4wSLCTby/yvZLCkuYLrCeubOPnvudtuLhlFKyA1SQYBc9+/VLIShXBCzyMktTM27AIf05gJoETQZ8Ra9pOnMjGFQkA8qJzRZieu3WEqzsdi9gbi6Z6/0iJpf5BP58tJByPUuydjHChb3oMAeDbv/gRrebVwRfV9qkkk/rnoB/5SDKJUZIxK+ycKZpWyMo8Refm/Rz3uglTbteBVG7eQpdjXSErdF7JRj5NimZQqE5xWDy0dXeDmFAmjavDqb/thVtZO6wNbgZv9bj/fA+dwMOD53srf+1ivD/HdjcwKZoFBa9oDfQ9D4GviJFJ0ZxyBq9vQlaqAlYAKr9OcTdXFl+4e2I9fjCIdWgLn2EjxHkPVRV4yEr9DJ7dg2fCP8zzosDDMMmQSRUYUkrRbGzR5ApeMWRFvXY4QSnj4EXnz90+wqOXt6zHdQ1F/hx6H+naCD2bMA/iFHePY5wvELxv/cJH8DVvuG/svtQVk2fSEGBtGR2lE+cN9XmZsKBbdIomML4DjV6jqYVv0dhEqxrHuhRKev/p7+Q0PXi+LyBqakIcFgOTornmHVkg+O/GTf15PgtwBM9hbSDlzil4q8NDF/v4+N/7mrHzcssCKXK8f46w3Qn0H0ijbpmwD6voPP96N/D04qepgnffuS4CT+Chi32m4FVff2TRpNnAF+4e68eklCWLZpxm2B/EeOP3/wJ+45lbYxU0Pmfo1xI89f8kBW8YZ1rBE8JYyJKsua3SJtCF189kvq3mqi8PqHn21jEeu2QTvGLIylFRwWOvJYQiXveOY5zrRZgW3LbKQTUW1v7ESSmttAitbE5Y0VlF53OQLk7wxlk9163g0XW3SUoGx7oIrLZoTluTkP/+8T2xUeTjtEEr7Bt0knmFjLs50F64lbXD2nDfuR7e8MA5PH3/uXXvisMKEAUe+pFfaQ/ltk1dk5AvsIUQEOw3FS3KecF1Uyvitd0ufuP7/iC+/MkrZgYvqrNoquj8u0dlBe8kTpFJ0+EXegJxKnFzf4B7xzE+8coB4jEplpzEkvpYJKmCWSQBYFShLEWBh1GaIZUmjEanaLI6gUng+0lEUxedS4lRIhurpIAi34M4w8Egxq3DYVnBC+2aBCJ6RP4DVmS+FQXaollU8JqgTsErFp3Tfkzq/KPrdxKRWVTIStZwO+uewVtXCMmqsC6LJhFnU5PQ7PWTlEJWmlmKHWaDqc/YnHOsFDxPf+zQTrgUTYe1YasT4L/8lS9b9244rAih76EfVS+auW2TvP+8lN0vzOABxuKrahSaq0tX8zRLIpXdGotwL/RwMkpxJyd4N+4Zgnc4SKxt0L7unaiv7+eP1/XQcRsqr4PgMDUJ6v9qBc/H3kmMXujDF3aEf5w2J2VWyEpBUaSi80lhJhzdnDB94pUDAMBjl/vW40WLJs3i9fPrQKu0oY9e5OP20RBxKnG+Nz3B87xqBS/lFk1GOCfO4HnN7mw3Vd4mga/Nx9nzaDar2Me4KtB1tyaH6NJB78OqyRKFN5mahOYpmqHnOYvmkkGXg5Tq5kZVYFfbkDEFz1067YVT8BwcHFaCKPCscnGOKgXPzKaVZ/AAs+CZRlniILWoTsHrRwGORwnuaYumIXgHuaVQz+DlZGB/oJ5L84R1Chr98aRgFKCi6LwmZCUqhawYi6aK8KeFdtY8ZIVts0TwMkXwprFoPnhBEbr/+Ls3AKCk4EW5NVWHrBQVPEbie6GPl+4NAGA2Ba+m6NlW8AK9H5N68OicTrJCygUpeE1n+YgIrEOtkVJudIomfy9XcXhJmuGf/PdP4HCYlIrO44bXUpJJ+L66ObaBb8mpAf952xShlN/82sSf57MCR/AcHBxWgtD3ShUJhCqCRzNQXj5bVnycFLxZO/22J6RodkMfgzjDnTxk5cbdE02eygqe2of9nNgRKawjRSFZEFnIil8TspIVFDweiEIhKymFrDAFb6qQlXEWzSwPWZkiRfMrn7qG8/0QP/XuzwAAHrloEzwhVHJqeQbPVvA6ua33pX1FrmeZwauzaHILa4+lelZZYTk48R0Heh8CT8xn0WxYmE7WzKYKzyLBT8UmWjT58a1iwfvRlw7ww7/4+/j1Z25pRVSHrDRkEVTT4iyaywW/3jflPGfS/P7bkEM6k3AEz8HBYSUYR/D412nRHTLiw8mPXyB4s3b6kS20juD1I9Uxd+twiNAXGKUZXj1Upd0U4V+0aBLB2zsZ5ftaowJRFYTn6edM7sErEw8KWclYqTEtCOMpQlaqFTzz+nEqp+zB8/GNb34ImQSu73YrVVIexEJETxM836i03dDHy3uzK3hVC9wsU4pTcQbPsmgG1deVIb7NLJq90J+vB6+hEkjWzHXM4HHSs0lhE4RsxYt4IvJpHnAEqOs49EVzi2bKgpc28D05LeDX+6ac54zd/NqUYzqLcATPwcFhJXjiyhaevLZd+dg2qxugGQZeAM5VOm3fy4mZP4V1kMMUndenaAJKjfus6yp1lJI0DwZk0Qytfd1rqOBpGyqvSaixaNIdYrJmRYUZPBWyYiya9Hxa4DVBlYInhFkcTmvRBIBv+cKHIQTwaGH+jkA1FABwlFs16T3hQTv9yNfzZbOFrJQXKVxdo30Bmlk0dchKwxTNbjQfwZtWwVuHipBZi9yVv/zSka3BogkolZlGKgNPIPS9qSyalNK7ie/JaYFt0dyME51K6XrwNgAuZMXBwWEl+JFv+fzax7YLs1dA0aLJaxKoP3E+Ba8TeAjykvUqcNXpDQ+cw4du7OOFuyd40yOqAw8wM3jaopkTv8kzeOWQlVLRef550aIZFnvw4hRZxiyaGYVtyMakzJ5xZB/n6te0Fk0AeOTSFr77D74WjxXm7wjconk8TOEJ9p4yEs/fn/Mz1iQUSY+p11Cvpy2acYrIt28wFOFri+b41+UK3r3c5jsLGit467RoZvzjzVsQ8jXuKhbxdO1kmdTvv08Eb4qQFT2X6xbpS4N9baxvPxYFmqcNtYK35h1ymBmO4Dk4OKwdpm7ALKrDglWQyIuxaM4XsiKEwJ/50kfxtievVD7eZwRPVXk8r4NW6iyae7lydy8neOEEi6aqSbBDTQh1ISsli2bCFDw2g6d6+JqdG04EOa/x8oCSUSrRj6Y/z9/7FU/WPsYtmkejBFtRoIm8DtIJPItoz2TRrEjRTDTBU5/3ddF5AkT2e1oEfXmSgkeP90IfN+cgXXyBNZ7gGdVn1cg20KbGMe/xnYxSdAKvcdovzdklmURKFk0ieFPP4G3me3JasGkKHh2Otmg6htdaOIumg4PD2kHWPG635DUJgFFOTIT+fCErAPA3vvYpvPW11QSPK0cPXOjh0lakLZoUsrJVIHj38tk7Inp1+9YNfJ2gWTeDZ2bg1P9VM3hR4CHJJJI0V/BYimbMirwngc8K8o+93N6YzKDgTQK3aB4PU12RALCQFabgdQKvdl5yHPwmCl7ILJoVVliOpiErZG3qRv7CQlbGqTfJGgle2sCi+c6P38S3/uvfauWCke/ytCOOWSbx1h94J37qPc83/p5Ez+BleqZWWTRFI4tmlklk+Yyp5yyaS4VF/jfgRNPvxjaGrPzq77861+/aTYMjeA4ODmvHTscuuAaMjZEW1PSQidD3rc8XDa4cXehHePBCTyt4tw6H2O4Euhsu0CmaiviNKhIvOb75LQ/jn3/r50MIlqJZU3QuCwpecQYPUH1yvqdIIU/RbGrR5PtpBdpYFs3FnudeGFgzeFSRADASn8/gAeo9mAVVKZr0OT/3ncCzQlbCuUNWoI8hk2bRPi2a1iRo1WcNFk3JXrLuvHzw+T38+jO319bTNw/sGbzpVrxJJnHrcIiX8qCgJiC7rVLw1McqZKWZRTNhi3ReneKweGza/Ckdj+nBa8dBPX/nGN/+Y7+DX/zoK+velVMDR/AcHBzWjmK4BgA9C0Vf8rSCl8/gLUDBGweu4F3sR3jiyjY+/rIq7v7Yywd4LQuMiQohK4Q6i+T1c1185dPXAZR75wgli2ZCFk3zPJpZOx6l5m59vv5Tdtbpi875t9D8zrQpmk3Qj3xliYSysFkKHpvBo9L0WeyZgLGZciTM9sb3p0nIiiZ4DYvO6UbBrEEr1gxeEwVvgsT0vT/1fvz0+2/MtC91SBsQoHSNPX3zghPYaW14dLzpFMSWrs+METyfFLwG549/j7NoLheWutvCa7sITfBaNoN3UkhkdnAEz8HB4RRAp2g2sGjSmrwb2J1pi0afKUoXtkI8df8ubh4M8erBEB9/5QCvu76rH9cK3qBA8BqoXmGByBK0RZNqD9IMQtiElhTEk1FqLJpECLPmqhsnolzBU4RRYpQs3qLZC339R/lolFjnmwfp9EP19XO92Qie75UXXkUFD6Bi+1RbYeuuKyLekxZzks3gAbOXnXP1pY4kkiUPGG/R3B/EeMf7buC3n70z077U7mODmoREE52WrBgZ7JqE6b6XyNo01tmEK3jFkJUG1xG9JvXgOYK3PPDruS1q1ziY341k0WzHMekbKS3Z31XAETwHB4e1oxhWAhiLptAWTWHVKBgFbzm/xnqRSenc7gR46n5F6N758Zu4dxzjdXl1At/vsoI3mWDR/heVyKKCN8pVNJ4oSgqesmiaFM0073lrGrJiWTQLiZppXpNQ1ws3K3p8Bm+UYisqz+B1Q0+/DzMreCx4hsAVDr4/J3Gi7ahCVB8v7dtEi2aR4M1q0WT7XmfP47bHhH38D3/uY/jbP/0h/Tkp0Iu2cXISWnda0g0heNMu4nkiZlNwMpxlBYLXxKKZcgVPTEx8dZgdm9YBSZeptmiucV+mwSw/Z5sOl6Lp4OCwdhDB4wtuUp+4chdU2BOXpeD1ckXpwlYEIQSevu8cAOA/vfcFAKgkeNSPRwgaqF51PXjEL2jNEKdZKfiDugDTTMIXipRk0pQjN51PtC2avBNP/cGnTq1FQlk0cwVvmOChC6Yvj89Z0vswS0UCACt4hlCt4Pk6ZGXcsXpNCV5mQlaAORQ8qfYzyZXUKnBbZsw+/uAL93A4NJalj760X3r+IsBPRR0BaouCN4iVGk7qOGAf37SKBh33dAqeUf104qsuOp+8nYRd356HRt/jMBs2bgav8LuxLQoe7ecmkOxFwSl4Dg4Oa0fVDF7JoukJS5GiRMWlhazk27+Yh3uc64d44HwPv/UpZW/jFs1Jhebj0HgGryLohBM+zxN5qbdZwDclv7UWzZwcxUk2cx1FHXqhr0nL8Si1aim4RZPeh1kVPL9CwUuYKsL3h2bwxhE8XzRb+GQFBW/WGbwsk/pab0Lw+KxXkkrrcyJ4TaP2G++jZWGsm8FrB8H7s//23fj7P/MR62uywfHVYZbjjtn32DN4zRQ8fQODQlbcondp2LQOSPpd2baic6fgleEInoODw9pBCl7VDB7xDSryJixbwetXhHs8nds07zvXxTn2dU4I+D422begoFQSDMFTn1cFnZBNlb6fUi81wWuoutUWneeWz1Ga1aZKzgoKHzmJUxwNE03yAXNOuqwm4dwcIStS2guVrILgkaI4mhAo0zRFkxZ+huDNNvyfNiB43KLJ1RpVcWE+/+hLy7Fo2l1g1c8hVfm032F/aW+Al/ftxEtboZxue/MoeGkmNTnzPaUqNkvRzPT3eEJMXe3g0Bx2RUj7T3QpZKUl9l7a73XUxJxWOILn4OCwdvieQC/0LZVIWzSZwmUnSPrW44tGJ/AgBHBxy1gDVeG5bc/k+woAl9jzm6iLdMxFgcz04HEFr0DwCgoehaLwkIUm4PvvFRU8WW0PnRemXDwtKXimzN7UJMxs0axIg+MWNoKaCUzyY60/b57XTMHTM3hzWjRTKfXNjNoZvNRW7fTHjCCkmdQzeIu27PFTUXdeiAQu2h66aCSse44wT9F5mh9vOsVxJxU1Cb6nrtcm7x23IHuiPSpMGzGPunsaQYTOb/h77rSAfmY24T1YFBzBc3BwOBXY7gYW0SAyQ4SDQkQIVHS+aOsgQQiBfujjPOtfo6CV1923az2XE6+rux39cZN9011sBQWvqgePzwUBBQXPy3vrpJnbaWpfrVPw1PxOhkzW1wbMCiJueycxkkxaCl7ILJrdBVg0AfsPP+8WI/TCQPfghcFki+YkIYVUwknq2yRkmdTqzbDmRessmhmb4fr07SOdWpos+LZ8WqGO1u3jaV8wpgVbKzDfnFU8g3IZV9YkeI0tmjELWfE9Z9FcJvjbsQncgq7TthWd0367a93AETwHB4dTge1OUAhZKRedc8K0bAUPAP7W//wUvu2LHtaff97D57HdCfAlT1yynsetkFe2DcFrsm9aoSylaKr/jUWzPINH5wBQxMPLLZW66Lgh+bUtpubrl7c7eP7ucek5iwARn9uHQwCwFTxm0Xzd9R184WMX8caHzs/0OnT9/NivPYvv+PHfAYBKArzbC7B3Ek+shKBTOtGimT9MNyJmT9HMi9h9r96iybbN5+u4AvSxXL3rhf4SQlYmE6C2zOCpagL7a3zNuIoZvLRKwRMCYUOLJo+6F0JMXe3g0BzzqLunEcWQlbaov6b/db37cZrgUjQdHBxOBXa6AaKgXHZNRMcXAoKtu5c9gwcA3/gFD1ufX97u4EN/56tKz+PE6+pOl329gYKnj3N8yMowLhMPruh5+byNlGaGpyn5tVQ7th+PX97Gz33opfxYFnueidC9mhO8rYgreKYK48JWhJ/6C1888+vQKXvPc3fwu8/dBQBLFSHcf66Ho1GK20ejhYSslGoSZk7RlPAEEAb1BI/PnfD5Oh7S8YlXDiAE8OT1nUYkYap95Opoy1M0FamqVvACT0y94OV2y6YwISuZPp+ep34umlg06f0N/fEWzRfvneC520f4kicuN943BxsbR/Bo5tMvW9tPM5yCV4YjeA4ODqcC3/c1n20RFpr54jN4XJAiBWiZCl5TcKWMWzSnUfDqe/DU5/dO4pJNscPOly9UiqbqrZvOosmfx1XSx69s4SivMliWRfPW4Uh93uEzeOq1uozwzwo6j7ePRjgaJZDSVkUI951XxPwzt49xjb2HRTQPWbEJ3jwpmp4QiMbY86wZvIIVlfbzJE7RCTxsd3wM4wUTPCuEZPwM3mkPWUnSrPTe0qfK7jjd9lJG1qbZB8BW8IKpLJo5wQu8sRbNf/2rz+Id73sB7/tbX9l43xxsZIWft7aDjiFsWdE5vQ+nfcZ3lXAWTQcHh1OBL37iEt70yAX9ubZoMgsjV+to/mzRxGMW8JmtC/1Iq45NVC8dslJ4KvEsWjDfPRrhQt8OGuGE2PdMiiZfFDYBfx7/lscvb+mPF32eKXzk1YOyRZPOG58xnBVE8O4cjZBJYBBnVsog4b5zPQDAy/uDxaRo5u8b9eDNqpqlbAavSU2CNY/H5jHTVHUlBp638JoEK0Wz5jDpnJ/2BZgqF7e/lrG5pGkJ6izHTe8Zn8HzPDS2aNINnlBbNKtf+2AQY7Bgsn/WwN/WlnChsaBLJfCbORVOC9pyA2mVWP/KyMHBwaECReuiUqj4DJ6t8K0TnMhtdXwdGNJk3566bxef9/B5PHJxy/p60aJ553hkJXoC9gwepWgChkzMG7Ly+JVt/fGiLZqkbH34xT0AsIrOTYrm/AoebYtm/Y5GiV7A8/Nz//lm1tqmBI8ep+t0ZgVPSp0gWzfHx0NT+MeKrBiLoJ/fJFl0TYLVgzdBwTvtC8aEpdAS6CaLP4NFc6YePKbgacucVnGntWiKWuJxHKcLD9w5a9i4FM1CTcIp/3HV0BbNDXgPFgVn0XRwcDiV0EXn+RpcCHveThednwaCx4hnLwqw3QlwMEgaqV4PX+rjHX/pS0tf5xbNJM2wdxKXFLyiRZO+hzrXmpIy/jxuW3zkUh9CqD/yi7doqj8/7/30XfRC3yKTFFrTXYSCl18fZDU9GiaVCt7Vna7u/RuXoknneNKd4mLR+fwKno9RYr/m772wZ1lygWKiprRm3wLfQ+CLhato/FRMsmie9p6qqpAVrWrMYNFMZlAWeOR7whIxVU3ClBZNUU+qB6P01L8fpx12B2T7z6V2f/jtsmg6Ba8Mp+A5ODicSoRFBc8TluJymhQ8j1U49MPpFLw6EM9KM4m9kxhSoqTgFUNW6PVILfIbWjTtagSbRD9wXlkXl2XRPBqleOr+3cqC+IUoeIXwmqNhWjmD53sC13eVijeuB0/3QzVO0cwJ3sw9eOo9iQKvpOD9wM9/DH/vv36k1H2nv5dZ/LSC53tWMfoikDZQMbjt8LSC5jPrQlboBsA0mEXB07bOzC46n9aiGfleXnRe/drHoxRSbobytC7wU9cWMjQOpaLzlhxS2oLfL6uGI3gODg6nEsUZPLqDTaDFf1Mb4rJB+9bv+NjOCV7TmoIqELGVUuLusQoiuVAgeFRkDNgKHs1qhQ0JJt/PIiEiZW1ZFk0AeMMD56zH6D3tjFHSmqLIS49GCUvRtI/pvnOK4DWyaE5K0SxYNGetSZBSwhfIaxJS67FBnOJolFqELa5J0UyzDIEnEHqLV/CmqUk4zYqROVf21+lMNe+LAAAgAElEQVT0hr439SKejneqGbzULFb5tRrmFs1JNlFu0RZC1M5FHk/Ri3jj3gm+96fer90BDgrzdCSeRmj7esuKzmk/T/Pvl1XDETwHB4dTCWPRVH9ohKguOj8NCh5gUj/7uUUTMFHTs4D34N05igEAFwsWTSEE6wM0ZIYUvKCh6uZ5QiuGxT4+CloZZ1ucBVHg6UXE0/fbxfH09W64uBRNwtEwYTakAsFroFZqi2bDkJWeDlmZbeGR5imaYSBKIStJJjGMU0sdLBa6c9JCCt7CZ/Aa2NTaoOAljAxzcBVt2vUubWs6Bc8sVuljlaQqrMfrYGbwPPhevW12kNuWm5DP33jmFt7xvhv49O3jZgdxRmBd+6f42m4KunFFfzva0oPXlp7NVcIRPAcHh1MJbdHMf0td3opwmZWIawVvDpVskSCy0I98bHXmnw/kISt3jkjBC0vPI5sm9eABRsGbhvySilf8nieuKIIXLSGtlMjPGx4sKHj5vixGwStbNGmBXFQr759CwZt0Z5sWe3SdztqDl2ZSWTQrAjaSVOIkZsdT6ElTKZpEMHIFzxcLT9FsYlNrg4JX19VHhxSMqRyo3abuwWv+/hNB4yE5QU7O+eP139/Qohkn+b5VP/5vfu1ZfOUP/goAVbMBYOEVG22HZU9uCRkaB5MY21KL5ga8B4uCC1lxcHA4ldAWzXwR/oPf9Ebw5fgqis6nAe1vL/Sx3VFEbJ59E0zBI4tmcQYPYLOIVsiKSdFrCt8TQFomPa/P7ZOXtsuvPS/6kY9RkuE1LGAFAL7g0Yv4qqevVR7vtCgRvFFS6lgkkEUzCsbM4GkFb/zrEs/yJyRgTkImJQLPq6xJiNMMgzjVC/5u4NkpmqlEJtVdeJOiuXgFr1lNwvQhCEfDBEkqca5fvrGxDKRkjSyFrBjFd+YZvCm+jZPClKmH9DsmTiQw5kejZNH8/9l783BJsrs68NwbS26v3nv1qqq7unqpbkmtDTVakRCSQEYCGyOD0YwNGCOGxTbjDwwevm+Gb0DY4/GADGMwMAyID8t8CIPAzDdCY0sItAtBo621tdTdarW6uru6a331Xr2XW6zzx72/u0RGZEZk5tuq7/mnqt7LjIyIjMy6J875nVPx2kOl4JW/aY9e7eMrl/vI8xwD+Vhn0bRhqnY3ArfI1M2ioxWyQvvpFDyNw3Hr28HBwaGAYk3CajvAsbZe6HHOsNLyrf60g0SgLJoeVlqLl7AzxmSCpaHgdSdXdaERNkOvN4+CR2S0KIi++I7j+Kuf/mY89/RqybMWQyfw8NxbViespPfctoa3fv/LaltMp2GqRbNwsGfqWDTlr2aGrJj9ZR6fP2TFTNEsLMTTLBe9fqm2g1oWTWPRk2aCKO5NiqbYHmPTahKkKtXgtf/tf/sS/snvfXLxHawJHW5SZdHkjRfxVbbPOvuRZuK9Y0x8H5BFk2YuL2yP8Klz1yaeX9eiSQSvalE8jkXp+zjJDILnFDwT5qm7EciFKjo/Ygpe4hS8CTgFz8HB4VCiSmUx8fs/8grcsdGt/P1+IlAWTR+n1zo43g3A2GLqImdMWTR7oVc6k0YKHme6By9SNQn1CRIR6qKCB2jis2x8/yvvxE3HWrMfuAAmLZoJerKioUhm6xC8uiEr5txWWDP9sAyUohl45TN4Q1PBC7yJkBV6XCKtnsEepmgGnFeSCRX932ABdnlnjCuyv3A/oJP47J/bNQnzKXhNSHWcalIoiLm45oKCRfO3PvwVvPvzT+HjP/P60ucHZNEsWaXneW6ErFQQPPk9MoxSDMbCzjmv1fhGhXk93wjkQl/rR2sGL1M3Uo7G/u4HHMFzcHA4lPALFs0yvOj29f3anZkIPA7GRPjLD77qTnzHi84svE3OpEWzH00kaBJ0yApT5EyFrDRQ8LyKGby9xA+/+q49fw3z+vE5Qz9KlRJWJHJNUjRnzZLRwk+UlE9WHNSFStEsqUkgpYc6/jqBN9GDB4iFJxEFf09SNMWf3pSeuHnrApZNRqchVoS4uui86ZxV1VzfNJjEPJVF94D+ToxlH+L2MFYqnIli0XnZS4+TTKmRVdcDfY/0o0SRQWfRtJHdcARPX+vA0QmOOQozvvsNZ9F0cHA4lNA9eAe8IzXhexzdwANjzOqPWwSMFLxBVDmPZoesiJ9pglf/K14sBrGw6njYQAuV1baPlbaP/jjBlpxpXOvYs10bvRDPu2UVz755ZWI7anus3sJHWTSZCEgplpTXBaVoim0UCJ5cmO+OhLrSCT27B8+IDjd78JJsdtR+ExRn1PrjBP/+zx+09ncegmcWfe8HUqWc2T+nXQ48kaLZ5NzNk6Jphqykaa6uYfpOJKI/rCgqjxN9A4NcAEWYxLAqAIa+RwZRqh7vLJo27BTNA9yRJeGoWjTpa+KoENL9gFPwHBwcDiWqUh0PK0KPoRMu9yuVMzG4f60flc7fAYWQlcIMXpOOQHOG70YCiXHr3VCSjxSbPC61vDLG8J6feM2M7dWtSdCPX8iiSSmaPp9QT8iKtzsWNRrtwMMgEmQvy3Kl0KRpbvXg0XaX1SGZqUWhsGje+8hV/PoHHsZrn3MKLz27AWA+JStOs329I09Ep0iIiqpGnusQpNnbbG5NNc9VmmuCR7Z12s9BnJaSM1IiqSezbNFLipz5ekVEBsGj68qlaNqwZvBuIAWP1OKjokoqi+bR2N19gVPwHBwcDiVo7qjYy3ZY4Xt86YEvnDFk2XQFrxUYPXjMvsPfZAFPd/tvNNAxrXcD9Foe+uME1wYRNuZMBaXrcdbCh4gMZyidn6uLLM/hMYZj7QCjOCuoYuLvO1LBaxsWzWJ8eyKVIK9ml1qzfRR/+tLCSMRzFC+m4CVpvvTEz2nQ83LlISv+HMmC88zgJYWaBCJ4RYvmMEpKi8/jNEPocTB506fslFsKXqVFUzxmME5cimYFzOv5qMyrTYO+1vXNjKMA+r5zCp6GU/AcHBwOLX7+u+7BS88eP+jdqIXAY3tD8HLgWj+uVPBCY1aRFoJjeXe+iUXzRlXwiOCtdQJwxtQ8UbE0vi50TcLskBWVfriggudxhuOyKmB7GOOUDKahhTkRvE7AS5UyStFsBVwp43GaLaVI3nytwOPIMq0wldkAGyl4Wb6vCl5cUZOgevA8HbBTd/GkCHcj5bJgrWUVFk35Oc9ywLyXEyeZlULsLJp7B5PU7eO9iD0DHYNf80bWYYErOp+EI3gODg6HFv/gZbcf9C7Uxk3H2ugsacFMYAwYJSl2xwk2SkrOAaAVaCsrCXDzKHi+EdJyI4FI63o3RJ6LFM04zXFyTgWvSYqmXphPD1m5sD3Czaut0vnHLBeq4ZokpNvDSBE8CiC5PhIWzZbvWeoPgcI6PFmTADRTlGbBDCHJcj03NzLUHt0H10zJ2tcZvIqQlXlUjSTN4Ht8rvAHkwxnJRZNullAqlqcZvC4/u6J08zoES1fpJPlctq+kR2zHyWuJqECN16KZtGieZB7Ux8uZGUSzqLp4ODgsAT8whvvwa//o5csdZucMWzuyg68KoumUSehFTwZstBAwfOPkB22CZRFsxOgG3oYRCk2p6SS1t3eLCtQmunHBiUBKYSL10d41b/7AD7y5Sulv89kiua6DIS5NoiN16AZvASBx+B7zFJ/1DYyI0WTSMISEyFStSiUBC8jC+HknFeTBRgpj/sFNYNXEbLi1VQ1nrg2wPN+7s/wxSevz9eDZ6h+pOACQCDnbRWBjsprDuIsNwhehUUzbmDRtGbwnEXThHlubwiCl9k3M47KMZmJwQ4CTsFzcHBwWAJ6reV/nXIGXO2LHrAqSyEpeJxN1iQ0sVx6nN+QFk2t4AXojxPsjhNs9iOcmJPg6ZCV6Y/L8lz17LV8jv44KX3c5Z0x0izHpeuj0t9Tiua6tGhuSYKXG7Nuu6NElJhz3XlmEtAk0zN4FLLSVBnTZeblKiMgbiiIeT9xckbmvGA6uV+zEKf7W5NQpTIWVY1ZpPPC9ghxmuOJa4O5UjRNm22SGTUJ8r1TCp4kW8Xy+DjJdApxRXdfU4umU/DKYaVo3gDkwrxZAxyduUI6986iqeEUPAcHB4dDCs4YHtscAJii4Bk9eKTAbQ0jtAOuKhTqIODshgxZoRTNtU6AbsvD1d0IwzhdQMETf860aGb1LJqkklQtnDOZokkzmFTxYK5jdkak4HG1WLfqEozCbCIpTQneH3/ycbzqLR8oXfCpu/4FBW9kkAizsqEuEpkEul+LtthQzszjzAsWzVm7Y6qV86SHEkFOsgyZkXZKqlxUtGgWCFrRollW7TCoEbKiUjTHiZvBq4BJ6m6EGTxdCXI0LZo3AsleFhzBc3BwcDikYIzh4vUxnnGyhxfculb6GNWDZyh4V3eraxWqICyei+3vYYRO0QzRa/nKmjavgseYiJ6fZbkzC6oDj6nkwyIoabLKwpnKWb61goJnhrYIiyZHwJkRFGIvPJMsU8m0QHOL5qNXB3hye1S6wDcVrizTCt6wJIq/iYKXKLvp/qycU0uN0T+nl9c1CbPsueL3cZrNNRsUG6QwzXWwD33Wabt0zRQJmrBo6pCV4vEA9ntTRT510XlqFJ3fACxmiTBJ3Y1ALujzaVaCHAWkTsGbgLNoOjg4OBxSfOeLzgAAfupbn41uRcee6sHjTFkCr85hQQw8fkOGrGz0QjAG3LHRxZNbQ/XzpgTYhM/5bIumVN4AIPS9yhTN0YyFc5oJm92xlg+PM2wNhYJnEobdcYJeqwWPl4d6JKaCx+dT8IhMjOPJ9E2Voikj+ZWCV0IimoSs0DlL0hx74ICegEkkk0wHlxRDVmYtImm/oyRbSMFLc9FfSNdRYKivJkErXlvComn3iGZ5Dg/6821aNKuuTVKXt4eR2n9Xk2Ajz8XnKpGVFgcJ8ztnXtD7fNRm8LI5bqTc6LgB79c6ODg43Bh48xuejze/4fmV5A4wCZ6+W785pRi9CmZH2o2Esyd6+Ouffh1efteGNSd5Ys4UTQDgfPbCJ8v14jrwWCWBIwWvauEsUhSFcrjeCZSCV5y78jkXSmFKQSGmgifj9jlTdr+mtQ1E8IYlIRtmjQC9lvnYPNdhKfPMou3Xos1S8CxlRvxZN1nQJNlzpWiSVTTV4TiAXnRHaWYRtOI5NS2adM+m+JjBlOfTz0gNvrwTqZ87Bc9GmmsL7UFyiy8+eR33/Ov34qGLOwtth77XOBOpzEdlBo++zg6aZB8mOILn4ODgcIRhWTQNheF4Ra1CFW7UmgQAOL3WBgD0jJ7CRRQ8j7GZREVYNMXfQ6+6B49UriqLplm3sNYNsDWUFs2CbZFSNMtIEaUx+oZFsylpov0blRA8WhQGHkduhqzEkwEjc82i7dNwU2yQZlPNK87gzVr00nbiNLPe97qLz9iYozTTWE2L5rSQlDidtGgWd9lSAEv2y7weKegJ0Am9DgJZrtOKm6jTy8Z//ptz6Ecpzl8bzn7wFCiCJ2eyjwpfUmFGR4SQ7gccwXNwcHA4wjBDVkyCtt6QwNyoNQkmuqaC12vNvR3OZxO8zEg/nFZ0PpoRspJmuUquFAqetGgWFTyPK4umqZjRNoSCxw2LZrOFOimMZQoeLao8adEkgkNk0LaLHl0FT9sd620nSjJrm3WPw1Q70ywzlGBt0RzEOpU1Ls7gGQqex8qtdkOjB69sntQkeFd2DYLnLJoWzBCcg1K7RnGKd332SQCLf1boa8GTs8ZHxaJJ30FOwdNwM3gODg4ORxh0V99jNkE73m2m4N1z69pUK+iNgJWWJsPH2vMfKxV6T4NZUD2tB29WyEqW5TDL2i/KOoWiauObFQjZJMHTM3hzKnjpNAWP9oHLuTGb4NnEaZ6Qlf1ZtJnntKzAmkjTLJXATDItKqmzYNZfpLkMWTGsvsCkgldm0aTPsrJoNkzRNInc5R2T4DkFz0SW57XrM/YK773/AnZGgrAvqnZrBU/Ywo8KX6LDdgqexo39v7mDg4PDDQ6awSNLDaGpBfHHvvnupe7XYQQteo93w4XUyloWzYK1rqiyEHTISrkykua51eX34AUxY1NclAcetyoQygiet0BNwrQZPCJtgccEQcnsx86r4Cmr4j5ZNG21zbBWGgolMJuk0rmNk8yalRTb9CqeNbkPaZpbBN+sSZgWklJq0SycwmGcoht6GERp6XtiEjm6CTFtlvTpijTL1Y2VgyJD/+995xHKKpZFb4aYM3j8CM3gqZCVht9rNzKcRdPBwcHhCMOyaBrf6E0tmk8HrEiL5kbD+cQiqsqjTZhF59SDV7ZYGs+awTOI4non1BbNwkLO90x1btIaqHvw5qtJGBspmhP7aNQkpFmuSE2pgldzwSispnr/9wPm4tA8PSpEpmZ0vFmT0FTBK5LhJMusPkUAiJPcVuCmhKyYKZomhlGqVOxygie2b94HWe+G6nrdb+R5ji+c3z6Q156GLNfhOwdFhh690sdzTh8DsLiKSM8XFs3Z33OHBcqieUT2dz/gCJ6Dg4PDEYYZsmIqeOudxUjMjYiuDFlZJGAFmK7gXdoZYRSnVjhKqKx1k88ZEXGa1oMn/6de7wboR6mM3y+GrHBF3pI0t6xKaS6JgsdUIMS8Cl6ZRbOqJmEY68JuQt3Xnec5i6JYLUEoKnizbGBEnmM5Q0eos/gmNc7nDGmeI8v063pcqCpJltkdg2UzeOp7AaX7PIxTHGsH8vmT1x6pduaNouPdoPJGxF7jr75yFW/49b/EI5d3D+T1q5AZKZoHZdEcRCnW5Pd903TcIugQjlrICil4rgdPwxE8BwcHhyOMl549ju968a147uljaiEIoHGK5tMBpOAtUpEAiIV21TrqDb/2l3jbx76KtBCyApQvvmb14GVGYTrNVW4P4xKLprZfxllmEwtp2fQYM0hgw5qEkvJyAnEHmk2k4xxF8yt45vEtumitC/OclYWsBKomYZY9V1s0Gyt48rjbgSeLznPrc+1LNXhmiqZ8DqsIWRkYCl7ZftH7bc7yrnfDA7NoUj3I9VEy45H7iyzP1U2Tg+IWwxnvZRMQUeJMzG8eFUWMPmeO32k4gufg4OBwhLHRC/Er3/0i9Fq+reA5i+YEKEVzUQWP8/LkwTzPcXl3jCs7kbRoFmanShbH02oS8lzYFLmqSRD7vT2MJi2anFtF3CYnorAPsyahLBp/GrSCN7mfpA75HkdmBLxQQmgyMYc2G+Zz9uuuvEXGSkJWPGXRrDmDlzZP0ST1rx1wlX5qzouGHkec5FMVvKTEojlRkxBpBa9MWSYr7kbPVvAOiuAl+zyPWRdpZvbg7T+7yPMcg3i63bYJzERczthMO/JhAe33Mr8rHrhwHd/yyx9WtvijBkfwHBwcHG4QLBKy8nRAN/BwrO3jjo3uQtsRaZGTP6e5sTjNhLWuODs1xQpXFrKi5mEoZEXasK4N4omFbmDO4KW5bXHMMuQ5FqpJmBqyolIm7ZqEYYmCV/dl48L+7wdsUlndgzfrGBLDotlUwaPH0GxtnGTqdQGoMvvBlJCVKM0nLZqF1x7GpuozeUB0PdoWzfDAahJ0t+DhYhxmiuZBRPRH8ibCqiTrCyt4hZCVo6Lg7YVF868evoovX9rF+a3FugUPCi5F08HBweEGgadsWVAzGQ4anDO89ye/cWGLJmflizmz4DrNcxVRTxbNaIpFs0zBSwvK0bq0y20NYmU3pSREn3OLSJprdlJjfI9Zc3pNMK0mQYeQcDXvZz7WJjnNFbyDmMEz3ypdA1FPqaHtxHP04NGxtgJ9zZg3bgKPixk8q8euZAavhkVzVRK8UgUvmbRornYOTsGj62a/yH5dZBnUuT6IiH66iaLV2AVn8DKT4B2hkJU9IHiPbQ4AHN1kTqfgOTg4ONwgoDCO1XZgze04aJxZ7yh1ZF54FUXnRIKiNLPi7cNpFs0pISu0ltUzeIKYbg0i9fo9SfR8j+kQkMwOWaFte5xpEmgslN/3xYsYRNNnmyKVojklZEXWJBCRIXVyLgUvNRW8/VlgpRWqobJo1pzBMy2aTRU8Om66RqMks9JxRaeibdEs2m3Lis4nLZoJOoFfeS0rgictmp3AQzvwECXlabB7DSKhh22xbc5ILuMyfejiDj728JXaj6frYFkzeKronLMj1YNHn8llkuxHr/YBHL6bCnXhCJ6Dg4PDDYJiGIfD3oAzVrqQoMV5nOYTRef08yLqKXji32uGgkcE7ZgkeIHHjSJsO70xkrY6q+hc7svF6yP8yO99Eu/+/IWpxzyuYdH0pUUzMYhumtl20boKXlPlqy4evrSLC9uj0t+Zr1MWslK3JkEpeHOkaNJz21LBGyepstUC5RbNot02MS2afPK18zxXPXgeZ6WVGUTk6aZCN/RU5+ZBqHh0jPtF9usil5/zKlW/KX7zQ1/Bz77zC7UfP4hsgre8HjwcqR48ur6XaZM9d9UpeA4ODg4OhwDFMA6HvYHHWelCgkhanGRIc22Pm5aiOZ6SomnOwwCCzHmcYWsYqa45WtgJ8ibn67LMUspIWTSLzmlf+mOh3NVV8MpCVuhcUH2EuSAaxamt4NVcK9k1CcsjFD/+h/fhl977YPlrVoTB5AWr7OySe30dNA2Y0QqevinAuW3RjFNRk0DOzaRA3iJDweMlFs1xkiHLgU7oIeDMKmMn0DWzIb9LOgdN8FSR9eFSUygtlxJkF8UoTht1DRYtmouSkUxayxlZNA/X6a4EHfayFLwkzfA4WTQP2U2FunAEz8HBweEGAS1AnYK3t/C4VvDyPMc77zuPUZwaCp60aMoFOClrZQvjaSEr5jwMIBZdKy0fu6NEkYUVIngeh0fzdQXlSM3gGSmaSmWSK6NZ/WbTQ1bkHX8+mdA5QfBqrhhNtXOZwRrXhzF2x3Hp7ywFz0zRNCyoxd+VbqciRbNpTQIgLZqG21oQvBzDKFVzmDaJlPsq3wtN8PQ2iBR0AqHglRadx7ZFsxt6aMl9OoigFTXfesgW21kurntWoeo3RZzmjY6RFLyVlg/G6n++qkB1KgCOVMgKHfeyZvCe2h4Z35FHhOUW4Aieg4ODww0CrgieU/D2EtwoOn/40i5+8o8+gw89eEktBKI0syya4bQUzaRawSumaAJCDYzSTC14aZEfFkrMyxU8rralbJRJZj2mDKQKARVF58qmxuQ29WOGEwSv3gJsr2oSYuPcFWHbKfXP6eW9mn1nSm0qpGjWClkhi6avyZSp4LUDjkGUyJAUqdqUlMLrFM1JUkokvRt6KrSliGLISif0tYJXouLuNVTIyiFbbIueSqFeL4MLJVnW6BhJee+EHvwKst4ERFgBHKkZvGWHrND83TK3ud9wBM/BwcHhBgHdeV13Ct6ewrRj9eUd9EGUIkps5aaORXNa0TkpAsUetLGRzqhDVrhVYm4u2nXICjQJlM+PCkSvDCb5KyN4wtbFVCS/ua1RnDYOGgH2riZBkOPy7dlkbDJkpXaKpnFO0yxXBL/OfBA9l1I04zS3ahJuPd7FE9eGVs2BSYbpvdIWTfsYAK36dOQMXpmtT9ckiO+SbqAtmtNuBuwVDmvISpYLxYuz5RCBOM0aHSOpsTRPSftwaWc0l/JEhBUQ85tHZQaPPq7LI3gD9ffDVs1RF47gOTg4ONwgoEAFp+DtLTxDwVMKWJJhImSl0INXXnSun19cTNGihbYDiNmssfFax4wUTRWgkuX2zJUkZR7n4DIQIilYM6ctBm3CluHRK318+699FJv9SO6nXOTK17cJYXObYvFxy1zUR0lWSWbN1zE5pa6BkARvxjEUFTwiRk0UPHoOYCu4d57o4vzWENeHcamCFyuCJ212ap/1a9Dxt3wuFbwygpch9DhWWpLghZ5K9jwIBU93Cx4uBS/NjHm1pVk0myh4hhrLhX03STO87t9/GO/4xOONX9+2aB6hmgS5n8va33NXtILnUjQdHBwcHA4UKy0fKy0fzzy1ctC7ckPDvFNuWhytGbxcL8ynKXjmTFtRGSmmaNK2okTH71O4QsC5ClARNq9JVYcIiu9xtYiM6yh4BUXus09s4f4nr+OrchGkZvCURVOXcw8LCl5dC5ldk7BEBS9ZQMFTM3jTX4OuDaHkZkqNa1KTQDN4AKwevDs2ukizHF+90sdqZ7LHLp5Q8CZVRzo2X1p2Sys/kgwtn6MTiv3otnx1HAcxg5ccUgVPpGgKIr2MBMekMLc5C4PYUGM9hjTLEKUZdkYJruyMG79+lutQH/40tmie2xyo721n0SwBY+zvMMYeZIw9zBj76ZLf/0+MsS8yxj7HGHs/Y+zsXu6Pg4ODw42Mbujj3v/1dfi795w+6F25oWERvFTXHEQFax6ty2mxXbRhZlmOKMnUHF2RZNGCkRUUPJPg9UoUPFIQCaS4qNoGw5Znqo5VMInnME6xPYzl8Yhjp0WhZxA8Cn8RIStaVap7hz2xSMtyFliZVNTqzOBZISsTM3jT98cMrkkMi2aTonOT4JkWzTtP9gCIa6ksOVHN4E2xaCoS6Atbb2m6a5KiFXB0ieAFHloV1/F+wFRFDxMoRVMEkiy+vTgV12ddayQV3ndDX83gxYXPdhNk8ngAgB2hkJVMKXjLsZWeu9rHM+RnzVk0C2CMeQB+A8C3AXg+gO9ljD2/8LD7ALwsz/OvBfAnAH5xr/bHwcHB4ekAkabmSs73EpwzFctNc3djw/oXV4as2AsFWiivdQLr3wRVsG28n0rBS+0UzcCYwUsL4R60XVPBK4asTFu0FxW87QERPPFzWhTSbpqkdRilinS0fK9+yErD/rg6iAyFtfw1TSup/jm9D5RMOWvRqwJBshxplqv0yToJh3TcpkXTnME8e6Kr/t5rTW43qrJoGrtM12EgexFLi85jYdEMPI5QKnlawTsAgkc9eIcxZIUzK1l3EdC1WfeaHxiJqD7nSNJcbWMeMmwWt/MlBcfsB5YdynRhe4TbN7py283DXLYAACAASURBVIfrmquLvVTwXg7g4TzPH8nzPALwDgDfaT4gz/MP5nlOk4z3ArhtD/fHwcHBwcFhYXhGqbGp2uk75/YMXtGiOU5SfOShyyqwhMIyigretBRN+t1qSQ8e1TQQdMgKzQQyFcU+i/SYz2/5HKM4wxYpeFIZJDuqsmimmuCNjECY0Of1Q1ZKbIeLYlagjL1I1I8p9uDNOgQ6txSY0VJWr9n7qIvOtYJnEvxTKy2tqoW+9V7SawKTFs207DE+hyfntooYJ5kipv/2778A3/vyO4wZvAOoSTikCp6wJ4uahGUoR03J2TBK0fKF1ZYqL4rqfBPQ8QBHqybB3M9l1VX05OfssF1zdbGXBO9WAOaE5xPyZ1X4YQDv2cP9cXBwcHBwWBh1ZvAofAHQago99i++eBFvetvH8cCFHQCzFbzJFM10oibB97juuEuLCl4qHyMVPD6p4NWZwVvrBELBK1g0U4qK57rvj0jrKEqt4JCDrEmIZ1RC2GEw+uf0Y79mDx4Vh8eS3DZS8FJ9rgieUYTHGMMdUlnQqk028fyiRdMkH/QY6kUs269xkqp9+Icvux3POX3sYIvOa9yIOAiIGzl28NIiaNq9NohSRfh9+V7GC5wrocaLvx+pkBXj3C9jZDfJMnRCSqk9XNdcXRyKkBXG2D8G8DIAv1Tx+3/KGPskY+yTly9f3t+dc3BwcHBwMGAufMpTNGXRufwfNigoeDsjMTdzTnYtrSqCZysjtK6wUzQ926LZIosmUwQrldZAQqQUPLEfvmfO4NnHUYZxgeBtSYsmPSfP9RwS/Zzmw0ZJOpeCZ3W77ZtFMzOK4Cfn8dQMXs0UzXiOFE3at1aFggcAd54Qs0Gd0BPvpfleFyyaHptUHc0qhcqicxmyYkIpeAdC8A5nyMqyZ/CaHqcgeFrFN7sX5zlXUZqpmwNHqgdviQpenos5xk7gFLwqnAdwu/Hv2+TPLDDGXg/gZwB8R57npZE/eZ7/dp7nL8vz/GWnTp3ak511cHBwcHCoA1vBM0JWDLJXNoNXtAie3xoC0ApetUVT/6yYotlr6Rk8WpjFmZ3EV0zRDDyuLZpy/+vUJKx1AhmyIuoRaKFPi1xSLMfFGTxTwZsrZGVJFk01I1kVspIrElMWsuLXtGgmRkJpmmWGRbNGyMqMmgQAOHtSKHhdKrc2z1ViWzRZiUXTVPkCObdVhEjR9KyfHWiKpiLNh0tNyXOhsHO+HLVL3YSoeZzDOFFJpzSDp9T5OT43SZorSzlnR6kHb3mKP30/dML6N2YOI/aS4H0CwN2MsbsYYyGA7wHwLvMBjLEXA3grBLm7tIf74uDg4ODgsBRwI1ChagYvzXUaXbEHjxbI568Jgkd9ZlUWTVYWspJl8DgzFndGD558fUBaOgspmoIU2GRn2mKQfrfaCZDlwJVdm+BRTYJJRIh4Do0UzdAvJxNl2JOQFSJ4FQpUXDEvl0sLKr0Ns0NWtEXTUvBqHDu9L9YMHi9X8NqBJwJzJKF/5PKuDlCZZtE0Uk2Fgldm0dT1DgRl0TzAHrzDqeBJVX9JNQm03TowLZqkxtLndZ5zFae64oQtSZXcD1gK3oI7Tdda11k0y5HneQLgxwC8F8CXAPxxnuf3M8b+DWPsO+TDfgnACoD/whj7DGPsXRWbc3BwcHBwOBTwjMVcZQ+eETdOAQjFzrknZih4lSmaqSAOPmc4u9HFD7/6Lnzjs0+pkJUky9UcWOhzRSgVwfP4BLGrM4NHgS4XtkcAjJqELFcF6oSWL9IXR7FWG0OP11Y57JCV5Vo0p83g6e4r26LJmQ6RmbWQV9cBpWhKJayOeknH2jYVvIJF8+yGVvCo8uLP77+Ab/mVj+DJbXFNKYtmieoYGSpf0eJJGCepUp4JdBzzKEOLgs7LYeskI6Xe48uxM85j0SQrIc3gqW3MoXbGhkXzKM3gmYe6sIKX2Cr6Ua1J8Pdy43mevxvAuws/+znj76/fy9d3cHBwcHBYNsxIdNOWSX+nORgr/dIgVaR8kYK3JgurJ2fwyKJp9+CNY7GIEwt0jje/QTQQ9cditi9JM7V/gcfUgtxS8LJCyEpNiyagy9l1iqZNgOg1OoEne/A02UyGdS2apoK3bItm9QxemYJHyYJ1UzTNonPkvKFFc1LB4wUF7yVnj+NNrzyLVz7jBDxJ0K70I6RZjic2RTD5hEXTUvC0yle0eBLG8aSCFx6kgneIQ1aYrAhZRnpjnVRbE8MoxcmVEIA5g0ef7XkUvFzNDLOjVJNQ0WE5D+g9oHTSw3ZToS4ORciKg4ODg4PDUYHHmVLIxqr6ILNI0jjJLMITeGwisfLCdaGErc5Q8KwUTZ9jnIqQlaJ1z1cBIbkqWg8Mi6buwTOL2mcvKKnMnfbTPEYASHO7JgEQyY/tgGMU2z14dW1spqq0yB30NMvxz//zp/Cpc5vq/GZ5OdmapuAxVl4aPm3f81ycXyJKtYrOS2bw/ML73A48/JvvfAFOrLQQcI44zVR1weVdEWVQp+jc95i0eFaFrNgzeJ5M3TzIGbzDZtHMcqGwekuqSWha6D6IEiNkhXrwFlTwVA/e0alJSDNYIVOLQFuYxQ2Qwzb3WReO4Dk4ODg4ODSAx0oUPMOiCYhCcHNdTtZKwA4nAeyahE8/dg1X5CK9NEXTEzN4cZYrGx5BWTRTQfB8aR2b6MGTpAAwqgNq1iSYUBZNIkDGiiLgHJ3AkzN4hoLXkOC1g/rJm2XYGkR49+cv4N5HNi0CXkZoxQweVRro18wlgSU1bCbBK5AQtc0aqowiw1Nm8ExQIiq9x5d3iODZFs08nyTMWsGb3K+oJEUToJqOg0vRPGyLbXMGb1FikRnpt00UvI4xg5dmRtH5nDN4pkXziPA7ZLn+PlyWRdOfonAfBTiC5+Dg4ODg0ACcM0W+tCqXWouycZJNWDR1yIq9eFM1CXGGN/3Hj+NX/uIhAHqhYhInUphGUaoIHUGQEKjQDbIV0uv6JTUJzWbwyhW8XJa6WxZNj6FNFs1cq1K1FTy5X53Am0uJIFAlxSi2358yS2paUWlA85S8LsErHGNYss3K58r9MuffOKsmeB4XpHkkFbxLO0UFjxa9+jm6DF0oeGULYrMHz0Qr8A5IwTucIStZnhspmottyySvdY9zGNs9eIkxgzfPrKRp0TxaCl6urvllWTSnfT6OAhzBc3BwcHBwaACP60WE3YOnFwJRklnWysDXqllxgUzK2OYgwu44weee2AagVRdeCFkBgH6UlCo7Qp3TCp7PWXnISkEpmGaDJCJXtGhGhhJZNoPXDjwM42wuBU+FjQTeQhbN3bEmeCaJLUvSTAyLZrEmgTGjU27GurmoiLVKtlkFUmZ9zybLVQjkop7eo0vXbYJXlvxJ++fzaguaSNH0Jn5OM6D7jUVsh3uJLKObG7PDd2bBJHX1LZqplaSbWAreYhZNdqRCVnJ1U2TRWgO6xkKPI/DYoZv7rAtH8BwcHBwcHBrAY2UzbPmECmZZND1N8IqPI2WM0ikfvLCDKNFBKXbIiljMDaJ0wqJJj00zkVxZVBZUD55hyxtXqIomxpUWTT3Txjmz9tP3eMkMXv0UzSTLwOUM4SJ30K+PRCn7KM6sYywjjWllyAr1/Ol/T0Nxf5sUnSdpBp9z61xOU/DIQkYzeFd2Z1s0I2XRZNY8qYlxkk2kaNKxHIRFU1sXDxfhSKWC5y2hB88ieDVIRZoJa243EDN4HvXgLXCuKLwJIAWv8SYOBGlu3JxZokXTcxZNBwcHBweHpwc4L6lJKISsAPbsXDDVoikWaE/K2oQozfDQxR1t0SxR8AZRWqrg+R5DnObI8tzqxgNMBU8vWnR332yL5rG2Dt5eafmKVGRqDsnYD5miOYxEDx4FvtSewUtzMQOz4B303VGFgleyzcSawdO/px48UmSbWjT1DF4NBS/NRfhJyftWBp+LHrxRrBNcARg2u0mLZiK7zhhjUgG09ytJM8uuWjyWg7Bo0vt12OxylLDKGMOiPMD8/qjzOaE0266h4JkzePN8buI0s66do1J0bgUkLdOiyet/Zx02OILn4ODg4ODQAJUhK0UFr2DRJOXEJBqhx9GWBIBSNQHgC+e3dQ9eYZYPEJUIQYXCEskFuujnMtIYPdOiqWcHi/tURJRmCDymFpIAcNNqy1DwciuEBJAELxQhK9TZ1ySIIslyBJKgLrKoVzN4RQttGcHLcpV4WVqToGbwZuy7PF+EJjN4ZjgOoV7Iik26Aj49RZOuHSrHNkHva7EmgX52ICErDcNH9gt0c8NjWJgMmfbTOsc5iMS13TZm8OIsM+yszfcnslI0l9Pttx8wZ/AWTtFUBI8rC/RRhCN4Dg4ODg4ODWB2I5khJcVFmam8tTyuCOA40fbKVsDBOUPocTy5JQgeY8Dnz2+XpmiaCl7ZbBaFuWiCZ+w3My2atoI3qwcv9LjqZuMMONELFalIczGvY+6n73F0Al8qeGJfmpC1JM1kih1f4gyeJkFlC2h7kZjhi09exwceuKi6zprUJHSM+TXfE8+t24NH1jBCsejcBM1TFkkXXV9lwTCkEgIUrW8/l8h+sSZB/OxgZvBoHw+bXY5ubhRvXlzZHTcmGrZFs4aCF0kFL7AVvEU6A02LJjtKISu5EbKy4OVppsw6i6aDg4ODg8PTBNyYtzFtl0Ui4lkKHrPI4Om1NgBdaB36XM1Pfe2ta/jC+e3SFM2WIniJpc6p3wcexkTwmK3gmSErxUVgakS0FxElGUKfK9Ky1gnQlq8DmBZG/RxS/AZRIhW8ZqXBMYXELHgHfUfN4KV2TUJJCXSSZcrWmuY5fuejj+DN77xfKni6NHzWjE+S5VZRudgmr2Udi1OhXNZV8AI5c0kpmoAMhOHTCJ6er/OnKXglFs124GEQH4RF8/CFrOR5LgN4mFS7xD6O4hTf9IsfxDvvO99oe7ZFs46CZ1s0aQaPApTmrUnwvaOl4OV5jjwHQtUDutg1EqsePCZt5YfnmmsCR/AcHBwcHBwawGN6UaxJWzqhgtlF52aKZoYzax0AoucN0Ivpls/xdXdu4EsXdtT2SlM0x6myUpkQCp5QzXhhBo9qEgKPqUXgeMZcmjg2QfDaBsEzlRwik6ZF0+NcErzUUBObKnhLsGhKBW+cZJYNtdSimUo1RtZgDOMU4ySVBNZU8CZf59zVPn7h3V9CJhWUTmj22Nnk9nNPbOFn3/l5bA0iAMC7Pvskzsv5S1O51M+fVpNg9+AB4lqj90IRPGsGL7fsupMETxCHsITgrbR8DOQ53U8cxpAV4sweY+Bcn+OdUYJ+lOKp7WGj7ZmErM5xEsGjay3wbAVvnpqEyOrBW9x2amKvZjfNlF5gcdWRnBaBnAF2Cp6Dg4ODg8PTAB4XBcB5ntszeBMET//d7MGLkgwbvRAtX8/f0eLkRC/E825ZRZRkOHelr15PbUc+bhiXWzRpRiqVISumvc8zkhXLbFxVC8IozdDyPUVC17qhFbZBKZN2qIyYwRsnIr2S5srqBiAkqVD9fIMYzwOawRvHqZqBBCpCVqRFU6SkCkI4TjJ5fPp9KFtAvvf+C3jrRx7BZWnN60woeGKh+NYPfwXf+Rsfw+/f+xg+9vBVREmGn3jHffijTzwu9isT5Ku2gifPj6ngmcSfeGLVDF5Z0TkFtpRZNHstH/0DIHiqB+8QqSl0LVPROZ1juraIgNWFeU3WuamhLJohpWgKtXuRmoQk1WEl5jEtiovXR7jnX/05PnXu2lK2Z4LeB22vXmx7tkVTV8ocNTiC5+Dg4ODg0AC04E6zHHYPXlaZfhjI8BNAdoz5HCdXWkoVI/K0sRKqvjlSn6xZPkNVKRadA5pIUk2CVV1AFk1p5aL9JlQFrZCCxzlDy+dYlwoeHY+oSbD30+M6lGV3nCgFL8/rxZirPrgFFTydopnNTNG0ZwUFsRUET9jwlEWzZHe2BsIKGiUZ4iwvKHgMnidI45986gnceaIHANgextgaRshzKFUsSTMETWoSZAqmpeAZ10ipRdOYNfQ9YcMz3xMii51w8vrqhZ6aa9xPKIvmIVJT6Jzyws0Lus6GDa2s5jXZJGSlO9GDt6BF0+zBWxKfvnh9hCjN8PjmYDkbNED7qFI0l9SDF3hMWaCPIhzBc3BwcHBwaABF8PLcIjnDKLWSJi1rZaEHL/Q5TqyEyqKpFbyW2gapT3aKph3eUQQpeJkxw1bcbzMav86icpykamarHXjCohloi2ZGRefGisLnHB2pLOyMYktNrKPipTJsRCh4i6Royhm8pEZNgpzB43JRR7bOLMslgRWPKyOoW0PxOmNpjy0qeGTrHUQpnnfLMfmcCNfl82iujWyifgkxL0OVRZPAS0hpnOiUT9q2adMkYtKuUvCidN/j81XIyiFSU2jdTzUJtGv0nTBsqOCZx9akJkEVnXscaZrrmzcNpaw8z5WKDVAP3nLON12foz2Y39SEbDkWzahg0TxMtuAmcATPwcHBwcGhAcy5JpM07I4TrLT8iccBkuAlNPeWouV7+InX3Y1//tpnAdB2uBO9UC3YSCkxrY9mdH3Zwt9S8GT4A4G24xvBAXUUvLEkpADw2uecwjc884S0aNo1Cbxg0aR0v51RAs9jyiJa5w57nOoeP9OW909/75P4jQ8+PPP5BDNF07KjloWsGOQqzTVpGieZsKBOsWhuD3ShepHgkXqZ5TkGUYKNXojQ50LBo+dJMpBI5bJ+yAr14KXY6IXyZ9MtmoLIUk3CpOpBi/B2WE7w0mwytXOvQYRnHtvhXkHXmIiahGI35iIWzTrHWQxZ8bmoSdB21mbEhIiMadFcFo+P9pDgKQXPq19HMg2mRbMsZfaowJ/9EAcHBwcHBwcCCSSpMYMHiOCTrkHwzIqCYopm6HO87nk3q9/TomqjF2pro1SfTGUs9KZbNGk2LsvswmxuFHUHXNyVznNh52oHfMLCaCIyCN6vfs+LAQC/8O4vTdQkVFk0hYLHtYJXpy7ACFkxrWafeHSztP+vCjs1LZqkXvgelwpebi1KTbJctvvbUomj+TSTHJnHMYhS9Fo+1joBtgea4A0UwZM1CcwkabN78LI8w03HWtjsR+UWTWOn4zRXjyElL84ydOCpcwWUK3h0A6M/Tqyk0L2GVpwPj5qiZ/DsFE2l4DW2aM4XstIN9AxemmlXQZrlUn2uvn7s1xfPU98ZfHkKnvos7cGNAT2DVy/ldhZMi6bvLWYRP0g4Bc/BwcHBwaEBuEFUTPKzO07QMy2aBWvlWC74aAbPhDmDRws2IidlKZpAuUUz9LWC5xmqk0kGfaMQeJxkatE+PWRlcn/HSSYjynN4rEBoPa6UyJ1RYoWU1LFoUrWCb9hJsyzH9jButHDWBK9Qk1A4VlrDkZ1SvLfidYZxCsZE/YB4bJlFUyRiqmTDoJCi6TEVONMNfKx3AmwNYmXtpGMi5ZJzpl5vmkWT5q7GcaqqN2ZaNI0ya/WepJMKXqdCwQPEzYz9Qp7rCo/DFLKSGxZNUZ0i/q1m8JpaNJvWJMibCb2WPYNnpXE2OF/0mdA9eMsLWaGbQXuh4BVTNBclZPT+LaOH8yDhCJ6Dg4ODg0MDKKtelmOcZjhmqBqUaAfYxKzXEv1haSaUomIEfctI0ZywaJakaALlC38iXmSb9Iy78ep5qi8qR5xmatE+VcErqGahz5HngpCk0g5q1iT4nKlzsTNKVA8eYJOJKoikR9kfJxds10cxsrzZIpFm8AS5ShXxKhI8WlDrOgd7bshMCS2dwZNKHL1nkymaXO1Lr+VhvRtgexgr5W+oZvDshEtgVsiKsJCNkgynVycJntpn06KZmiEr4s8ozfDQxR11vICu8DDRK1yb+wFzgX2YQlbsFM1Ji+ZeK3i7UYKWz9V76HviM2l+jpucL2VNLFg0L14f4ars6JwX+rO0fIKeKQVPErxFaxLIqurxCYv4UYIjeA4ODg4ODg1ghawkGVbagsgkWa7upgP27Fw39JHn2spXjKDXFs2WlT4JTEnRLLEqKgUvtQmeqeAF8u9xKmyLPUnEqhaVpkqp90Ps4zhJVcqkeby+N5mi6TdQ8FJDwSMydm1gk6EoyaaqJHmeY3ecKOvWzijRZLZwrLQQVnUOWWYs1DMVhQ9UWDSV1VISvEKKJmdaTeyEIqhmaxhjW3bhDY0ZPCLgXkFlKwOFQERJhlPHWmBMW9UAgJGd2NjpyCizpvfkPZ9/Cn/7P3wEF7ZHOryjxIKpFLxo/wieucBepDJj2dAzeHrGEljODF4dFWowTtX7QfsB2MSyyflSCp5h687yHD/+B/fhX73r/trbKcNezuDRuVIhK4taNFPbonmYbio0gSN4Dg4ODg4ODUALfVqsmMEqloJnLMyJ+F2TC/oqwrTRC9XCej4Fz1M9eCapsuoSSMFLhYK3MkvBK7NoSnWHEju9Qk2CL4vOzdek81E7ZKUwg0fnjlSAn3/3l/ADb/t45TYGkSCfJ3otAIJgrcj3IU6KCp4kePKufZpry+pYKnh0eEWCGqeZqrTYHZP6VabgSUtd6GOtE+L6cNKiOYi0yqiDUKZbNGk/iTiWWTTNXU4yrcjS9fH4tSHyHLjaH+sZvGkEbx8VPHpvGFvcfrdMZGq/hHqdFmbw6PvhV9/3ZfzpZ87P3J5JZOsEewjHgH2dAUWC10TBsy2aXCaDXtoZ4epuVHs7ZdCBRXtH8Og7avGQFa3mBx4/VMmtTeAInoODg4ODQwPQgruM4PWsFE39HFLJrvXFQqlImIi4nVwJwTlDO+Cqw81UxqyQlSkzeNTpRqTKit03bHmm6lh1t79cwePqd1R0bjoJhYJnqwt+A4KXZJmMKdcLrC1F8MR5f3xzgEdkGXwZiCCfOiYI3vVRrN6f4rHSPpk1CabVjhbxjGGiIoCqDgA9F1WWokkWzW4oLJpbg0hZO0nBG4y1ykhv2XSCp9+Xlu/heDe0FLwyi2acaJWQ/iQL3jBKFUEoXqOAGbKyfzN4RPA7gXeoSqdpVzw5t0mnWBedi2vhHZ94DH/wN4/N3J5l0axxnP3ITu2lz/V4bgWPbnJQD564bvpRitGCxCzaQ4vmhIK3aE1CmiP0uHAlcHZkUzQdwXNwcHBwcGgAWnDTIvdY21TwvInHAVrB2+xXKXg6RVNsx9fKhfFQxpihvpSlaIoC8iTN4HFequCRBYssZEQoqqLvp1s0JcHjxVJ1btkUfaNGoV6KplYgSdm41qcqAq12bQ+jyk42IlQnV8Q5vT5MDDtq9QwelavbM3jicV5J8MSWQfB2yaJp1VkIq+n1ERVTixTNfpTial8SK3lMZtUGLdi9KTN4Jplr+Ry3b3RxcqWlflamOsYyqZP2DQA2JdHsRynGcYp2wK2ZSgJdx/uq4KVaUTxMi+3iDB5d1+OCRXNnlOChizszuwOb1iT0x2mpgmeSqGYzeOJ5oaHg5bl4r5sGxhSh1PC9UPDy5YasJKnuiQw8dqhuKjSBq0lwcHBwcHBoAApV2ZSKknkXveXTYH5uWRZJzSKbYVEd6bV8tAOutmUpQIWFNpG40h48ud1hnGKD6Z6zMgWPFIaZKZpJZhWsm/tPM3jFzj1zBg+QxKlRD54RsiIXqUU74zBOEadCYTDfAwJZIk0FrysJyrQZPM6YCo+h16FjI9uaCVLhAE18ijN4HmdKUezKkBUAOHd1IF5Dlof3o1SRqLKAnCLMOcx24OHXv/fFlnJcZtGM00mL5iYRzSjBKE4rKxCIIO9ryEqmFbzNHI2i//cSZNHUKZr2DN4oFnUldK6u7EbqWixD0jBkpajglc7gNQgIodc3i86TLMMgShfuPRzHexiyUlDwFiV4cWrfADlMtuAmcATPwcHBwcGhAVY7YnF+ZUcsilcMBS/wuJzbSFFM0QR0UEiR4P0P33AnXnP3SaWaWGEthcVs6HNgXB6yQtsdRKmI55cP8TxTXStX8IpzaYRxmYJHM3ixTOxktiU14FyeCyaj/7k6H3VmWsyaBFqkFi2apCpsDaKpBI8ULUoDDT1ebdH0RNqnqViMZMgKIG1rhf3fHur5pEHJDF5g9BECQuVdk9fQk1tDcSxxqkrS6WYAPadMqSWY2235XG2XYCa+EhJZxUDHCwCbcsaqPxYWzbKAFeBgZvCI4NM1F2cZWnz/OviqkOUGwWNsIkUzTnNL3f3yxZ2pBC82wj3qJDf2xwluPtZW/9YK3nwWTbrBQzdiOGPKJr64greHNQnFHrwlWDSJLHqcHapgnyZwFk0HBwcHB4cGWG2LRfTVPil4elEd+lwtNEz+RQvjTTWDZy9Qz6x38Jq7T6l/dyrqFug1ANueRyCCN4xSeFyTA7sHr5zglSl4eS6Kk6dZNKkmwVRViFASUfBk0AhQbwGWGCErRL7MkJU8zzGIxeLTVNBMFGfwABEBH3isOmRFqm2DwkKUiLeZlkgwX59eM/Bse6xF9qVFE7ATOcmuuaJm8CavoyKKBK8I+rVl0UwzFYVPz6dreSCJZpWCF/qCIPcXXPA3ARF8upYOS6phcQZP9eAZn6PLO7pegGooqkCqXTvwaqlG/UKKJpF1U8FbxKLJGFPv8/Jm8PajB2+x7SVphtCwaB6W660pHMFzcHBwcHBogNWOWFSRgnfMUvAYQkl+iot6wCR40//77QYzFLySn4vtiucNotTqnjMf25aPoXCQlSkhK7RYrQqFGScp8lwkhprHS3N+phpFRKXOginJMgRchKzEqShTv2YQqXGSYRiJfdselhM8PYOnCV7L4wj8MgXP7sEbFmoA6PTNsmgSaaYEPnHs3ArEESErofr3qrx+rkgVradm8LSSUoWiRbMIIqZ20Xk+UXROFrxhlGA4xaIp9s8rVfCSNMPPv/tLeOuHv1L53HmQGMTH/PdBg4gFY8JGSyTavHlwaWek/v7gxd2p26O5u27o1bZomko/kfWxYYOssl2Xv77dg2dedosS2+Y48QAAIABJREFUsz3twZObDJVFc7HXsCya3tG1aDqC5+Dg4ODg0ACk4F3ZtRUXANICOLkwp3m0rYqahCLM+bUij6OFTFAi7ZgzeGbwiTnHt9a1939a0Tn9bKImgQheTAoeCj143DoOQZzmU/AAsZimcwfItMdouoJXnMEDSGHlEzN4cVpQ8AoKFb2XZVH9W8MYjInZTOqHCzymlVxPHzsgSO+6YaW8Za0DwLD8Fmbwmlg0y+BxZgV8xGaZekEF7o9TOYNX/Zrd0J8geOMkxY/+/qfx2x95BO/+/FOVz50HRMaVgndIiqfpnJJCS/82SdXF6+I95UxYNKchVjdT6oXJ1OnBm0fBo2uKWwQvmxkSMw1KwduDkBW6HpY3g5erz67PWaM5xsMER/AcHBwcHBwa4FhBcTEJXuBzdQfcTtGcbtEsgkI6GMNEmiHNIpUreHZPXpmCR+Ti8o5NUMuCFGhhNitF0+N2TQK9Hh2H7xkKXs2QFd8IZkmyXKVoAmIRSzbKrWF5R5eewdNqGVkMp87gMTYxc2SGrBQXutuDCKvtAJ1QK1uenEEEqAdPE8R2YM/KnV4Tc1SXC4SbCHPdkJVWheomwjIKM3gVSaxiFjBVKm8ZVlr+RMjK+790Ce/70kWsd4PGBd+zQPuuFLxDoqikhRm8tDCDB2gF77mnV2cmacZZrsq1ZyU3RkmGKM3QC+1ZT0CobfR5bZI6GqU2USoqx4sErdA5Ge+FgldM0Vzw8rBugHCOPD9c/Yt14Qieg4ODg4NDA/geRy/0NEEqCVkB7KLzli/skmQzrKvglUXkFxMQrd8Z2+WMGWEahoJXIHhV1QGAXtRVKXhRmiLLc7DCDB4tNrWCx5WKVa8HL1el4/Tv7WGstrs9jFUy5LQZvG7oWX18OvilegbP95ilggDariZm8OzX2RrGWO8GaAVcVWf4hkXTM4h2N/DAGFNBPQBwiyR4pODR/paR8yLMOcwq1e2Zp1bwwQcuIc9zPVPpTV4XgEhWHcWZlQJaRK/lTZC4S9cFkXnJHceXTvDIckf7dFhCL+g65rIvja4LkwhdkgreS86u4/oowSVjJq+IJM3gc46A85nEjBJwbQVPq/ekds5j0aTvkCLBWyRoZbynM3jiT9WDt4QUzaLCfVhU4yZwBM/BwcHBwaEhVjuBsjgeK1g0zZhxAmOiNqD2DJ5c5JfFweuQlbIUzfLuOZMkELkgxagdePA4K7VokiJlkiSgmKKJSYumXGx2zBm8hj14vqeDWdI0x7VBhJtXBRmiwngAlnXTxM4oxrG2b82TkUVz2gweZ5METyt4dmCJeP0Y650ALd9TFk2fMwS+tnnRsXdbmrzR7B0peEXLLy0upxE8z7qJUE7K/slrnoEHLuzggw9espRK2jcTA5miOc2i2StR8OjGxem1tiIfywLZZ6lb8LDM4NFlwJmdrloWsvLC29YBAF+90q/cXkzXfI1gDzr/PeNzSe9lnusbK4tYNIv3lhaxV+4twSNiKr9fFkzRTDLbogkcnmuuCRzBc3BwcHBwaIi1TqB78AoKHqkjRfWtF/q4Pqqn4HWmKXi+PaNl/85Q8AxroLmQbwce2gFXi8/AY6W2RQAqRa9YQ1BM0fQYswitbyhWtK+0v/UUPApZYXI/EgyiVKldmwOT4JUreJQyaJIVIuBRovfh7feew6NXBmq/xcyavS1yMrISi+bWMMZaN0TL56omwfcYAm4oeAVFE9CzkErBUyErWvUEphedmxbLKlL2HS86g1vXO/jND31FkaWgwqI5iNKpPXiAuBaKM3hbgwhrnQCr7WDpCZvFkJV+lOBNb/s4vnB+e6mv0xR0HdNNgWxKyArNWU5Twaif0Pf4TItmMQEXsNVY+v5oojzNsmguouDRtkcL9umVIcvta3pRO2WUmAre4bqp0ASO4Dk4ODg4ODTEajtQJMC2ALJSiyYgFu70nLopmtPm7MpqEsLiDF6FCrTeCRXBC32O0OdTFbzeBMEzi85tiyZn+tiJ0PgNCF6W5chyWAoeqVu0ULYUvIoUTSqCNpWt0LdTNLcHMd78zi/g9+49J/bT46XnXCmhjKG4Zt4eRFLB47pLzJrB01ZT81pZ74TWMV0uKHj09ta1aFYpeIHH8QPfcBafePQazm32redNhKxIi+Y0glcWsnJtEON4N0A39BAlWaPZr1mgkAvap8c3h/jIQ5fxp585v7TXmAeqB09e22lJyMqlnTEYAzZ64r2epmCRah1wVpoEadYskILXrejL1BbN+sSkrOjcxCIJmGN53GmWL91iqxS8JRad0zkInEXTwcHBwcHh6QOqSgAE2aHFBVkAgck74CZJqqvgla3t6bll6YomcaR+rrLHrnUC1X1Gc4Nli0G1kCzMZJkpmnlud72ZwR8dK0VT/GyWhSo2UvGIgBAZVQqeEbiyXaHgDcYpuqEnKwvEdijllBaZV2T33Fcu7er9LFHMmGHRnOjBoxk8315sk0XTVC/NUAyahSxaNHW1RHWYjvk6hGk3De7Y6AIAru7q91y8hn5+6HMMpYJXVXQOiJTPSYtmhPVuqK6TYo/gIkgLCh4Vy3/6sa2lvcY8MIvOGdNVFFGSKdv2petjrLR8pcoWrb8mYjmD53tsoibh/ie38a2/8hF8+rFrAKCUYlNZ962kVrJo1icmZtE6HZeJRSyaJuldlk3z8c0B/sUf3qfUTAq3WpTgmRZN+nwdlmCfJnAEz8HBwcHBoSGoKgGQBE8uLkLPTNG0n2OSpFkpmtNm8FoVEfeATRxNYlEkCWvdQC2EWj4X6lOS4Rf/7AH8xRcvqsfRPFXRoklKly4612TUJA1lNQmzeqpISfAMiymRn9OK4OkOwqoUzX6UqBklSoUszuDRTCQRFlP1NEGHxJgIyvnp/+dzuLI7RpJm2B7GWOsEai5RnB+tPpopmp2CRbMTeFboTejxiZ7D6QqekaI5heDR9UR2Vt+bJP6nV9uGRXP6DN4gSi2r6pZS8MTrEAFZBpJC0Tn1Hn7+/Hap6rxfUEXnTBadGymaNOc6jFMca/lq36cSvCxX12eRmFE4C93oKLvxYnVdBs0DaRTBUz14BYK3iEUzMQnect6zv37kKt712SfxyBVxc8bnRLSXZ9Ekm/VhCfZpAkfwHBwcHBwcGsJMQQwNghf45T14gB2IUGavNDE1RbOJglcygwfA6mGjZMlxkuJ3PvpV/KePfVX9bnc8Oeuj9sPjyqLJubZomq/VMRIhdchK5WED0ARPJFqSRVMQMT2DJxb5Z9Y6lTN4g0j3hFGFgLKjytcgRYtQpeCpkBUOfPDBS3jHJx7Huz7zJD53fht5Djz75mOFigqh6jIGq4/QvAZecGYN99y6pkjfzqhQXF2n6Nw4537xjoIBOg9Ejsp68E6vtXF9FCPJ8qk1Cb2WjyTLrbTIzX6E491Q7X9/iUErsVLwuHUMUZLh/ifL5/D+9DPn1eP2CjpFE9YMXpRmVg3GsXaAtnyPp82xJbIaxONsQjEiwkw3XMpuvJjfKV2VONokZEVaNHmFRXMJISvA8hS8gSS5u7IOhUuivUyLJn0+XE2Cg4ODg4PD0wCrE8EqOtlylkWz5fOJu+NFKIvmlCCVWSErpgI2oeCVENSntkeI0gz3Pbal7ljrGbzJBX8r4KoHj7rAAFtVKpvBi5IMH3jgYmUnmFlcTPuvLZr2DN4t621sDeLSbe2ONWEickDvDwVhbPZtghcUZvDo7/QjcwH54Ycu46MPXQFjwKuedXIiwTTwJ8+/qbj8j699Jv74R19p2SFNIq0sr1MUPCJ1s2Y66TyQ2llMCQQEeabzMbUmQf7OnMPbUhZNsf+LBHIUUaXgAeU2zceuDvAT7/gM3rXEGb1Hr/Txqrd8AE9tD9XPzBk8LmsS8jxHlGRYafvqmllpawVvGrmJ00xWg/AJYkaEmWo4ytJtzeuWfr5Mi+YwWqwHj/ZvvKSycwrz2VXdk+J92AuLZhOifFjgCJ6Dg4ODg0NDVCp4nt1/ZoIW2bPm74DpCt60kBVrDoxpW+REyEq3qOBxPCoj3IdxivufvA5ALCQ5Q+lMFtk6sxxWiqa90DR78MTPP/rly/ih3/0k3nv/xYltAkYnnacJkk4jJIumVvSiNCu1vg3GiVrotk0Fz7Jo2r1kpuoJaIVEz+DpWbp7H7mK9z9wES84s4aNXmgreNKiqRVU8btuCVEmBdV8PfGc2RZNesy0UBSxv1LBG9gKHm079DiOd0M1z1RVmg5oEkpkI0oy9KNUhayI3y1fwSPSuT0U2275XM2kmXj0qriOq5TdefDQxR2c3xrikcu65oCcxty49vNcKHgtn6vPzLG2r25WTJ/ByxF6Yl60SMxIrSLiXJZuayeqNlfwkjQDY5M3Nehm1iLKW5RkxnaWY3ckFZMUPFLfF1bwkkzdOKHPiQtZcXBwcHBweBqAZvAYE4tsawavQsEjsjFr/k48dnYVQpklb9Kiqf9uokzBu2qoWZ98dBOArBoI/VLFseV7hRm8SQWPFrmmgvfUtiBr/99nnyw9dqUkGDUJj20OsN4NsN4VaYTXBkTwhKJXXMxnWY5BnCq1iRS8sFB0frWg4JmddYBYnAPmDJ74819+y7MxTjJ87oltvObuk+J8mDN4MkWzGJTSCyetrmL/xH7axdVipmia2kvnZ3avolTwaAaP29fQase3VNrpISuS4MkFNvUQrveMkJUlKni0YKdzRK/38rs28OlzkwTvsU1RebFMiyYdqxkuQwqex/WNmEwqeKHHlT2Zzlcn8KaqYEkmFTyPT1g0+4rYkZInbry0C3OfBGXRbFSTkCPg2l1Af55caQFY1KKZqptiS1Pw5A2GHUPB840003kRpflECJGrSXBwcHBwcHgagFI0xZwVs1I0qXC3yM2IbMxajANAJ6CQlcnfhZ4mTZO/KxK88kLrNUmU6Dnm806uhPj4V4ngJaWqEx0H3dVnrLxUvWvO4MmfX5Wq2fsfuDiRxvh9v3MvfuE9D4h9NoJKzl0d4Nb1DgJPbOdq357JKxK8UZKKwueWHbLSUiErYsG22Y9w+0ZHvSdmZx0g5qcAm7w+9/QxfN8rziqi/WoieIUUzdCfnIGssj52SgieqWBWgRai0xQ3c7tVFs3VTmDZ/WaFrABapaOS8+PdQP1umQSP1CwieNclcXvp2eN4ans0YQd9XBI86pxcBvqFGThAp8GaFSEpETyfoxOKc0jXUDv0pit4SS6svZxNKEZ0jHRey268WLOvpOAlTWbwMssVQNf8iZXQ2od5ECV6LnHPFDz5PmQLWzQzNUdNpNmlaDo4ODg4ODwNQAqeUu6UbZJPsWjqGbxZoDvwZQEbOmSlJAzEqASwZ/AmaxIIgacVyF7o4ZuefRM+ee4a8jzHbpSUBqwAQrGiBacV6FKiJJhJkld2BMkYxRne90Xbpvm5x7fVzzzO1LZ2RgluXe+AMYa20dl3Zl0qeJK4fPCBS3jo4o5akGsFz5PHKlJOIyNF80SvhbtO9tS+WwpeixQ88bOf+fbn4S3/3deiE3p4xV0b6AQeXnr2uHwNfY4DSU7pmIkAVCl4dJ7MGgVzrrEKdM5nXVMtX9hFtwblISur7cCaD5ym4NH1sKsInjj3x7uhet6ehKzIY9waxgg9jjNSvaWEVcK5q8tX8AaFGTgAau7TM96nPNchHV15k4ZU4E7gTZ/BywQx9D02oRj1C+EqZTde7BRNEfDTxFqYpJlK0AT0DaoTPaHgjRdILI3STH1njuIU3/Pbf71wjyGpmjtj8T7Td9CiCp5p0aQbTMvsddwvOILn4ODg4ODQEGQ3ak0heMWAFFKTljWDV5WaSEoShQ4A1SmanIntkIJ3Zr2Dl991HJv9CF+90sdgnExUJJivQ4qEsGiKnwcGmVQ9eJ7el6v9MTZ6Ic6stfEnn3pCLZTHSYqdcaIWkqbFkfbN3Gboc1Ugfa0f45fe+wB+8Hc/gV99/5fVQljP4BkKqzGDd3U3wkYv1ATPmptj6rXobXjN3afwotvXAQA/++3Px//9fS9R53uiB8/jtRW8MoumaWutAp3rWQoeYwzd0FOkp1iTsNaxCd60mT6VlCnJjrJoGgrenoSsqBm8GN2Wh5PHxHt/uUDwyKJ5fbg8klkMNwF0Gqw5g5dmpoInZ/Ba+hqcnqKZy/d8MmRFpWiOtVWzeOPF/Kz4HkdQsp1piNLc2gZ9Xo/3AnA2/3uaZTniNFeuh81+hHsf2cQnH520117aGeEzj9frN5xI0eSCaJfN4KVZjl94z5fULO80xKZF0yl4Dg4ODg4OTx8oBa+QYjhtBq+RRbNGimaVfc/8vVLwCoEspOCpvieD4N2xIcjOhe0R+rIsvAymRZNzpuxiZSEr5r7EaY6NXogf+IY78ZcPX8GffOoJAJOJlr6h4AHAbcc78nU9tW0Ki/mp//IZ/MYHvwKfM2wNIq3gSTJi1iQEHrNSNDd6Ie5UBE+TKjM8p0xJe87pY/hbz73JOh963zle/7yb8N+/9DbrnJSlkQL6/V6xZvD4TIJH56dd45paafmK4IWGyswYEbx6Fs1bVsX7QERKWzT1DF5RwXt8c4AfffunLItjXdDiumNYNHuhj1MrsiB+RxO8PM+VRXOpM3iSTPQNkqNTNPX7m+U5olQSPLm/K4aCN7PoXM6IFpU3NXtnKHhFNdj8rPhSyW/agxca26BLvhf6aM9QH6eB1HL6zjm/JZJIt0ren9/60CP4wf/08VrbnUjRZGLmuIzgPXJ5F2/98CP40AOXp24zz3OhpBZugLgZPAcHBwcHh6cB1AyeQewAIPCZWhwU1bdmISt+6TbM1ywrOhfb16SksgevW24xPbPeVqrY5iDC7lQFj6vFlTl/Z/dx0Qwet0jS8W6AH3nNM/D1z9jAv3rX/Th3tT/RSWfWJIh9sxW8TuDh5EoLL7xtDV//jBP49e99MV5z90lsD+NJBc8niyZTM3h5nmNzEOFEL8S333ML3vjiW7HaCSoIXukpmDgfBN9j+LZ7bsH//HeeK/6tahJmWDTNHrwaCh5td5aCR6+hi85NMsqw2vFrK3hr3QC3rndUB51p0Wz5HJxNFp1/6KHL+LP7L+DLF3dn7mcRSWqHrGS5OE+k4F0xrputQaxCN8wZvAvbI/zQ735CpYg2BZGJgRmyonrw9M2NLBNWxtDj6nyqGbwaBC+UqnWRUAyKM3hROnGzwPysBBTW0oDgJWlmXRcqMVYWtdcJWfnz+y/gze/8glVbMpYzd3RT7IlrkuANoonnX94d49ogrjVHR5/xHaXgCUJWdsj0PVWc+S0izXLkuf580Hdsk7CawwJH8BwcHBwcHBpipWC3LLdoTn/ONHicicVyyQK/5duqWxEmAfRKVDVA300PiwreWkcTvH6EwZQZvGfffAxfldUK9Dpk+SSUKXgAsN4N4XGG//MfvBCDKMV7vnBBKXjm/puWsVslwSN1qRN6CDyOP/2xV+N3f/Dl+HsvPIO1ToDtYawW5L2WbdGkkJUoFdH+UZJhoxfiBbeu4Ze/+0XWLGHocYssz4JJsoqEmmYgq9TQspAVspNOQ90ePNo2EQyThH/Ts2/CK+46UVDwphPGrzmzii/KKo2tQSxqAUIPjDH0Qn8iZOWJa/MHnxBJMY+xG/pqNsycwSNV8bbjHUvB+8Sjm/jAA5fw2Sfq2f+KGBQUNEAQTYDi+elnwqLZ8rkqN1cpmqGH8RSCl2Q5fK9cedMF59oqWlTwvALBE9d5s6JzO2RF/NlreYKc1ujBe88XLuDt957DR798Rf1snIp9Jlv7NIWVSF+dGU66iUCWbp9zcC7eg//wvofwN49cVY+tS/BU2Tt9L8rPbeoUPAcHBwcHhxsfvsex0vItgkeVCUEFKaBQhDqLcUCQgbL1/WufcxN+6luejWedWil9nqXglRRaA0JVYKxMwesodW+zH2F3nFaqTt/83JtAN+rpUE3VEBAR66+5+yReePu6RVaPy9e47XgXx1o+LmyPFMF7zbNOyn3mlkp5q7RoEhkqI0trnQDbg1gpLbroXFo0PU8d68XrYh6HCC2ByGor0ARvVjE9oM+7Z9hV1Tbl+1h1LtUMnvH7H3nNXfjlf/iiqa+pLJo1FDxz2+bNgd/5gZfh773wTO2QFQB4wa1reORKH7vjBNf6EY4bqazdljdhxSTVZh7bZCyLp80bB/TZW+sEFsE7J8nDPbeuYXecKCWIrq0L12fPYJVBWTQNZZLCPDiDlaKpQ1bEOVytsGieu9rHf/zLr+rjTDKpvE3OkZnEjv5dvPFivqdEFJsoeJHcb4Kp4LUCXkvBo/P8a+//slLxKBCJCJ5W8MoInvjZLCIGTJJAj+kevP/rAw/jXUYNi37/ZhC8zC57p+8y14Pn4ODg4ODwNMFq29f1CPKOOWNTis7D+goeIMhAmXK01gnw46+7u1TdM7dvqmbFx3qcYbUd6Jh9g+AFnlg4b/Yj9McJVirmxl569rhSAnU5sm3RDH2Ot//wK/Ci29ctknncIFU3r7Xx1PZQLdT/7j23ABDpg8qC6HOckM8hMlNGQta6Ia6PEmXb6pWErND+XZR9fBQDb54bgBQ8SjMtPQUWzHCbImYpeNqiqRftZ0/0VAVDFczzMwumpa9M/a1r0QSEggcAX3rqOq4NYnVTQGzHt2bVAOCJBYJPkjSbIPu0rydXQovgkTr0NWdWkefavkfE49LcBG+yJiHPtUWTPqdxmiHLYYWslM3gpVmOf/GH9+F//69fxFW5/4rIctGDZ9ocTWIHCAJUtGjaCp6YX20SDpIUCB7dpFghi2aNkJXNfgSfM3zy3DX8tVTQSGGj7xEi2WUWTUrDpfdtGoo2YM7F99zuOEGS5bhubGN3bM/rVSE2Ap7En3pu+KjBETwHBwcHB4c5sNoJLAVMkz1Ndkx0G4SsAMLSVccaWIQiJVxbNMsCWdaM/aeFDNkgN3pi4TyMJ5UCgu9x/K3nnBKvZVo0y8r7YC9ATcXnlrU2Llwfq8Xhd734VvzxP3slvubMqlpoUUUCYBC8EjWMCCctIumcF2fwzMdsSKtfcT9D35saslJEyyDWRdDPqmoSVCBHBZmuAp3retUbdkLnxO+N93mWgvc1Z9YAAPef38bWoKDghR6GS1TwlHWR2woeIBRiqt0AgMeuDnBypYXTskKBXo/mBOdV8MpqEtJskuBRx5tF8MgmHGqb4+/+1aP47BNihpFmCInI0mfRJGfFGbxBWciK8Z6K7Qgr8nvvv2DZFatQadGkkJWaCt7f/prTCH2ODz0oAk1IwWv7Hlo+V+ft+iiZUCpJwZtF8PI8n1Tw5PcdkXnzWtuV1uBZCh6dc52iKS2aLkXTwcHBwcHh6YFXP+skvu7ODQBCzXqNVFt0iqb9+CYzeABZNJsTPCKaZkhHsQcPEEErtK/dUKhlN68JsrPRC9WivIqUAMDrnnczAK0Qcl5dzm0TPK34nF5t48L2EJv9CMd7IThnePldG2CG3ZPsmYAmeN0yBU8SvKe25b7Lc37zahvd0EOv5U8QvBNFi6ahiim7a423rBVUE7zbjndwrO1jo6AWqmMqUfDqIGhi0SzM9xVhns9ZhPHm1RZO9ELc/+R1XBtEON4zFTzPIkKDKFHF9PPN4OUoBs6Q3fnksdbEDN7ZE11li6TXo0X/xet2pUJd9FW4ScUMnjxdVCUQejpFk0JWqAdvnKT45T9/EKdXRQqoUvBkPL9XktyoFTxhO+1HqUXIAVifl0AS4iTN8O/e8wB+88NfmXmM0yyancCrVZOw2Y9wy1obp1b0+0IEL/T5xHV63SBhSZopYjdLaRsnGYqci+zhpAyaBK+YuFmFKLEtmjr59+hZNJt9kzg4ODg4ODgAAH72Dc9Xf3/jS27DG18iIvGP90Jwpq1ZBD2DV0+lObnSUlHsTUBEwywKLyMdp1fbasHzj7/+LF5x14bat+PdEJ86twlgOul47XNO4WtvW8NzTx8DIBZZVemeZiLouqH4nF5r4/LOGJd2xhNki7ZFpdYA0DFCVorQUewjcKaJyhtfciu+8dmn0A48RYDJojkxg2emaHpNZvBkoEyJ/fGbn3sT7nvzt1R2F1IpdlOCxxjDSsu3iuurYJaol10PHUNhrrL/mq/7/DOr+NBDl7EzivGKZ5xQv+uGvhWBTzcKgHkVPF0foI9FnKdTKy18pEDwXn7XhjofRQXv4sIzeJMpmozp64NUrsDneM7Nx3Db8Y7aF7JoXt2N0I9SfPfX3YG3feyruCLJp5jd0xbnOMvQgXhPTAWPSNBqe/Ja8biYQaNZvjjNcWV3jLXu7OsjSTNL5bVDVjg2+9NJzjBKMYxTbKyEOLESqlRcsmi2fLGdbX05YGsYK7u2rbhNJ2JlShwFJBGZ3zG3VztkxbZoHuUePEfwHBwcHBwclojXP+9mvPcnvxE3HWtbPw/lIrWugveWN96DeZYVZseZWV9QxP/xXfco69GpYy2cOqatiid6oeo3q+puA4Q68a4fe7X6dzFF0wSXnWt5bls0T6+1keXAgxd2cOfJrvUcsuWVKXhlBI9mwZ7aGqIX+mrh7Xscp9fE+xH44mcXro/Q8vnEXJwKWfG5IstNahLKyBObQnzFsYjnTlNLq/BH/+zrcdvx7szHmYpPUHINtnzRu1dHDQSAb3jmSXzs4Su457Z1fOcLz6if91oentzSq3hK0ARsxaYu4jRHIINriMAQET51rIWdUYJRnIIzhie3h7h9o6sCPej1iGxc2F6M4A1KevDMzxl1xbU8jm+75xZ8m5wnBcT1mma5Uraecaon940UvEwmx0pSkeqQkiTLsdr2cX2U4IktcT6LNyYAce1FkEXnnihWvz5KatVDFC2azFDwWjUsmlf74jg2uiFO9EJVQF+m4B1r+dgZJ1JtE+fBvCmwM0PpLaa0AtSDx9TsXRlh7I+nH8OERZPUVEfwHBwcHBwcnt4QQhi/AAAgAElEQVTwOMPdNx+b+DljDP/bd7wALzm7Xms7N622Zz+oBBTX7zFdM1BGOkxCV4QZgtKEdHhTLJq0T0meY8Ow9N0iidf5rSFecva49fj1boCffP3d+PsvulX9rFPDovnk1nBCQSXQ4u3c1QFO9MKSxEsdshIqu22dmoTqcz0LNE9Ytc/TQPNws2AqeEGJ55Qxhm7gTS05N/Gj3/QM/OCr7pwghJ3Arkl4fFOQvVvXO3MpeGmW614ySfDMkBVAVCWIbkPgjo1upYJ3ZXc80fc2C3meW+Emar+MkBXa3NiYwSuCzhORzDtP9MCZIJ9ZliPLdX8doOshaP7v5LEWro8SdT7LCJ6yaMo+yks7I+s8TENcsGieXBG9hid7rVohK9f6sdqvEystPHBhBwAQyZqE0OdqFvZZN6/gvse2LFJnpmrOUtrIKku1KEwmmZqf0+1hjDzPwRirnaJZZdFskkZ6WOAInoODg4ODwz7hH73ijj1/DVPB0zN4zUiHaZVsYhtkjFWGrNB+JFluWTRvNohs0aLJGMNPvv7Z1s9aUxQ8Wtj3oxQ3r5UTZFrEPnBhB9/zdbeX7qN4Ha5eq17IiiTWU5S6KnzbC04jTjKcqdjnZcB8H4OKfey2vJkBKwTGytW+XqEm4YlrA7R8jmfetDJfTYJUtsR+c4yTzApZAURQCW377AlDwRuJRf61foxjbR87owRXdiOl5tbBWCpoHmcYRKkiDVlJyAqlZJYTPHv283gvwEavhav9sRHPPxmyQvNjJ1daeORyXymiJ3qTN2jMcJDA47gkZw63DLJThSLB+9bnn8Zf/i/fjLVugHbAMUqmkxxS8E6sCAXv6m6EPM8V6W35XJ2DZ52SBG9gltTrv88KWRmocxJiexiX9n0mWa7qJIgw7sxr0XQpmg4ODg4ODg4HCXMGjxY8TVUlU8FbaUDwzqx3LDtlEbQ/6x1TwdOPL1MliujUIHhAtfJolod//yvPVu6jqeDVCTNVBfR1ElkKOLnSwg+9+q5as37zwrTaVhH+rkxMXARUk5DnOdIsxxPXhrhVzqLNY9GkkBVAL7i7RYK3M1Yl53dsdNGTAUVUeh+lGZ53i6h2aDqHR2Ti1EoLaZarmTJy7XGGCYtmWUE9XbdPSQVvrRPg5EqIyzuRiuEX1SZ2yAp1OpLiTlUQxXoPQL+vvuwNJEKTymCWaZhI0eRMvWbb1yEreZ7jTW/7ON5+7znr+TT7drwrZvCiNMPuOEGUalWTbpg86ybR4Wmqdk0UPKpIoPefZka9wueHQnZ2SxS8zX6Et7znAaXaAZMWTfrzKFo0HcFzcHBwcHC4gVCq4DWwpAG2ktZtEN3/Jz/6SvzE6+6u/L3Hmei3M/bneFfXNZQtWosgFaDMotkOPEW0qjrnaNH20rPHS+2NVsjKHDUJ8ySf7gcoQCOUfY1l6ATeEgiehyjJ8NaPPIJ7/vV78dePXMXtx0Wy5VwEL8sMqzHNKuoUTUBYLx+72kfL5zi10gJjTL5egk05f/d8SfCaViUQKbhptWX9m+ZX7Rk8qQCVKHhE8CjcZ70b4uSKUPASQznyjZAVQCt4pySZefxatUXTN25OBIXrsKx3zsQ062onFDN4eS4I+0ceuoyf+9Mv4L997in1GCJ4J3otpS5e3Y0UIQ49PYP3jFMlBE9eG53AmzmDRxZNInhlCh6granmDCUpr+/74kX81oe/gk8/dg2AIP5Fi6Z3hC2ajuA5ODg4ODjcQLBSNCvubM/CxpwKnoh5nzKDx5kVsAIIqx9FxhctmmWYpuABWsWrspaudsTP31Si3gFmyIomi7VCVoLqFM3DAHofp4W9nFgJrQqLeUDE+t2ffwpxmmFrEOOukz01L5U3TIY1lSX6k95bul6u7AoF7/aNrlJz6PU2JbGhpNemCh6RCSJYpOhRyAozZvCmKXhUhfHU9gg+Z+iFnkqbJAWPis4BQ8GLJhW8XlhOxFVqrtH3SCjaY5M0w8OXdtS/I1nTUIZ24CHPRZXCF86L/r7bjnfwL//4MyrAhXosVzu+ulFztT/WKZoBR1t+nm5ebWG17Vv7tDWIwBlwy3rbUvDefu85/MHfPGbtj5pLlK9TZUWnfTO3R+8nKalfvriD81tDvOotH8B//dyT8vzZ87SxU/AcHP7/9u48Oq6zvhv49zf3zqrRzGi3NjvG8hLHixKEs5gQZyFxFhIasgFNQ6DlAGkLPQRKetq3Led9G9ItlAOlC6EEmtLSlMUtS6ABWiAhibMROyFOHBwvkjfty+zzvH/c57lzRxrJkmVp5PH3c06ONKORdDV65Oin30ZERJUU9GTwvIvY56L+JHvwTsT2SUn5p2F6oiYvHS9npkXnQDHAmy6Dt7E9jm/dtRXXeyY/es03g3cyQ1YWg3k+Zrq++96xCX9248Z5fR5zXnYdGsYtPZ345l1b8eHLVyMW9iNXUG6f2mx5h6yY740pvw35LcRCNg4NJbF/IIkV9cVporGwHyOpLAZ1Zml1SxSWT+Ye4OlyQBNgmQDBO0VTZtGD52bwRlJIRPwQETTUBNE/lnZ7v2xvBs8MWUmXZvAODian3afozXRODuSHJ7L4+//Zi9sfeAIA8NUn9+OqT/8ER/Xzkc0XEJgm+Dc/c6lMAbt7R2D5BPdcfTYyuQJeOz4GAO4eSxEp6Y00WbGgVQxKm2qDSEQCk3rwsoiH/YiF/CU9eF95fB/+5cnSctDxySWa+rInr/cY8UzPND/CJtg7POJkQvccGcOz+weRKyg8u38IAErWoziDfZjBIyIiogoyv1xaImhLhPHXt2zGVee0zOljeAO8cqWQJ8snUjZD1OoGeLMp0Zx+iibgyeBNEwCKCDZ3JqYtU/QGeGZwymx642yfwCdLt0TTBF4zBfttiXBJT+TJMIFkQQHdnQl0dyZQVxOYMtlytrL5gmeBty7R9JQNX7ymCf/1fB/2HR9HpyfAczN4OsBrjAbRXBvE4eETLzt/Zv+gm40z2aLm2tISTW8Pnsn6ps2ahBkCvL7hlDsEpiEawHgm7wYd5YeslGbwktn8tH8I8S46n5xFHE5m8cSvBvCTV47j+FgaP39tAPmCwkt62uVMJZqmLDqVy2N37zBWN0exstFZb9A75ASIA+MZ1OvsvJvB8wR4Ac+QlYaaIBIRf+kUzWQWiUgAtSG7JOPWN5Sa8j3zThYFihk3E5+aigBz1sbSOTdANt8/k8Hbc2QUuw6NAABeOz7uPH928WfY8gmHrBAREVFlFRduO7+k3HheB2pDcyu7iwQsBPSOuBMtvZ6LcMBy/+ruZX4ha5xDD950JZpmF95cege9SoaszKFEU0QQtK1pJ1RWmgnwZppyeip4l2V3dxZXgsRCZjfdzAM0JssV1JSx9d6s8gcvWYXRdA7JbB7LJ2fwkll3RUJdTQAtsZC7OsD4y0dexndeKPaSHRtN46bPP4avPumUBboZPH1Gze2xVM7Jkls+mKc0lZt+TYI5r8ls3h0yZM67WZ1QWqJp1iSUZquA6UuZbU8gbH7+TWnuUDLrBjU79w24vWd7dICXnalEU/+bkszksat3BOvbYu4wJbPzcGA84/6BxrzsH0sjnStmNRujQbTFQwjYPsTD/klDVjJIRPyIBm03gzeSymI0nXMmjXr64Mz3wHyeyfs+O+udazMlweOZnDutd0y/r3nOXzk6ht29TtmpCUa9PyN+y+eW0J5OGOARERFVkbmUFU7HKR8LnNLyTAC4/9buskNYrt7Yil+/YHnJFMzpdHcmcP3mNmxoL7//LXaCDN6JlKxJmONzGfTP3INYSWYwiTc7sZCfJxq03WEaAE46g5fLe4asuBm84vd2Q3scb1nTBAClAV7Ij+FkDv26N6w2aKM1HipZwq6Uwhd++hoeeqJYAnhgcAIFBbysAx+T8Sn24Dm39w9MoC0Rgm35pkzRLBcoeddPmDUhZhiJCTa8pZVuBi9tslXFoG66TLe3B888V2ah+nAy65an7ni+tySDpZRCZoYSTROcHhicwLHRNM5piyMW8qM26JTHAjrA0wFr0LZQG7TRP+5k8Gw98OmuS7vw8AcvAuBM2xyetAcvEfY7GTwd4PXp7KBSwNHRYhZvIpND2G+5wat5us3PXnvCCfBGklm92sLp+/M+n4dHUvBbgoHxDJ7aN1Dy9Xqzn7YlyLFEk4iIiCqp2As2v//F10UCJcuxT4XzlteVlNEZ3Z0J/N+3b5xVKWQiEsBn3nnutMGg24N3CjJ4cxmyAjiZjoXOkJ0sk1k7mTUOc2GCgU0d8ZJg1wy3meskTW8Gz7ycvKvv965YjbMaItjUUQz6zVqG/rG02xvWURfGwcGkO+jl2FgaqazTV2buM0HF3mNOb5lboqkDBJMB2j8wgRX1TvA0ZU3CDIvOzbUBxRLDPk8Gzx3Nr7NGZj2Bd+/ddNNmLV9xVYcJUtoTYQQsH46MpNxy1e/uOqw/ZgB7joy6E0FPVKL59OtO1u+cNmciaVsiXAzwJoolmuYaTYBnno+aoI02HXwlIv7SHrxkBolIANGg3y3R7B0uBuMmCAacyaI1Qcs9a5OnaCYiAdQGnSEu5mOZDN5oKodkJo+hiSzeuKIOgDP9dJUOhIHSP4LYenfn6WZp/itEREREJ8XN4M3z//DNseCsMmpLjbnmuUz/9CpO0ZxbDx7gZPBmmlJZSQHbVxJALBSTXdvsKc8E5pPBUyX7HMN+a0qW9Nzldfjxxy5Fc6y4wHxDewyZfAHf3XXYLWnsrI8gnSvgmM4GmZ1yQxNZ9OoAwmT49h5z+rHMmgLTg2cCvgN6aidQ7Dvs1ysZyu7BC0wN8Mx1mdUN3im03jUJZuCPCWxPVKJpW8UJuvU1AcTCfuw54mQkl8VCUMoJkq/Z2IpXjo65ky5nmqIJAP+75xgAYL0O8Nrrwjg0mEROT0v1ZhYbokFdolko25OY0D2SZm3B0ETWKdHUPXiFgnKDbaB0+ulEOodIwHafd1NGbgLteNjvluiaAG+ZW2Kbc59vk/kFgOs2FYcuef9IY/t8XJNARERElWX67SInWaJo/OG16+c9UbESEuH5ff1uiaZtzbncNWj7luwUTcAJvhY6AG1LhHF2awxXnbOs5H7Tg9c/nsZ93/tlSankTLKF4vAP2/LNumz46g2t6GqOYjSVc1dzdNY5AdmBQSewM8vRAWC3Hv/vLTkcGM9gPJ2DSDGDNp52hqL0j2fcklBTivpinzOso2wGz3Of6RM1mbh9eriHbYmbYfWuSTCZdDNcZrohK6U9eGagSQCJiN8tOb1uUysAYHNnHOvbYpjI5LGvXw8XOcEUzWf2D+Hi1Y3u97ItEULvcBKDupfOm1lsqAm4Q1bKPR/xSAAFBYymc8jlCxhN5ZAIO5k3ABjL5ErOyOQMXiRgIeI3JZqlGbxY2O9kcFNZtyTTZPDGMzn06cxgd0cCtSEblk+wfUPxvE4p0WQPHhEREVXSpWub8OX3bnGn3J2sruZo2UXgS108Ynrw5lmiac+9RPO3Ln4D3rll+Ul93sVQE7AXPIMXDdr47ocvLhmwAgC1IeeX8R3P9+LzP96LB376KwBOr5tZZVBOLq/cpd22T0omaM7E8onb71nvZvCc8sADA8mSlz4BdvU6wZk3qNh7bAzj6TxqAjZCfh984mSA9vc7gaEJ8OprAmiMBtxSy3IZK9szIdP7R4hIwMLjr/WjLuLHumWxYg+ezhqNp/PuHyvMy2kzeFYxwDP9dPU1ASTCfhzX2cVrNrUiYPmwZWUD1rQ4gelu/bVPdzbWttTi2k2t+PObNuGf3vMm9/72RARDE1nsH3ACxLrIpAzeeNrp7SvzfJiM2ou9I25WNxHxu+dkLJVD73ASrXooS0kGL5NDTdCeWqKpX8ZCNmJhXaKp+/la4mbISs4NFlsTYZy9LIa1LbVY1RR1f86roUTz1HZPExERUUXZlq+k9OhMU+zBm18Gr2QP3iwjvJt7Ok/qcy6WSKByUz5ty4do0HZH0n/7F334wCWrcMPnfoorzm7BZ991Xtn3847vD9i+OQ3PuXZjKx5++iAueEM9AKDDZPAGihm8ZbEQoiEbL+pJir3DSaxqqsHeY+PYe3RMBxMWRAQ1QRvjmZyb+fMOdVndXIvjY/0AZi51zOZz7h8hACezNDCewT//5vloqg1iOOkEYma59kQm55YimpfTDVmxfD6IXtXhZvCipaXWa1pq8e3ffTM66iJuGeiLJwjwaoI2Plfm+9OWcIKmh58+CKB0ampjNICB8QySmbxb6ux12bpmJCJ+PPjYPty59SwAToBnfv7G0jn0DaXQlgjDtsQtqwScoLc2VCzRNO/jXXIfD/vxq+PjbolmQ00Atk8wlsqhTzkfa1kshHvfsRGFgkLA9qGjLoL9AxOlJZqW77QcssIAj4iIiKrG5o4ELlnThA26T2iuvENWIgEL79yyHFu7Gk/lJVZMLOwvGfax2OJhZ4DGslgIh0dS+Mi/PYtUtoDv7z6CQb0o2yuXLyCtpzACwIe2dc1pUbrPJ3jwvVvc2yG/s6bj4KCTpds/MIHl9RG0JkJ48lfOJMXeoRSuXN+Cg4NJJ4OXybtBZU3AxkQ67waIJQFeSxSPv9YPS0+MLCfst9xSROP+W7sRCxUnjprgIu/pwTN/rDhRgOf3FUs8/Z4STRNQ1gZtRIM2VrfUOtcDC63xEJ474Cz4nmv5bodelfCNZw9hTUu0ZIBSfY1Tgtk7nJy2J/FdW5bj7/5nL146PIKWWBDb1jbjWb2+YVRn8Da2x+GT0hLNiYxzhkxPordHE3DOeSzkx0gy5+4RjAZtJ0BP5zCWziEe9iMcsLDKM+l1RYMT4Hn/CGL7hGsSiIiIiCqpIRrEg+/dgoYy+/Zmw/RH1UedyYv33rhxSrnh6eqPrluPj1+1rmKf35TfffKGcxC0ffjZq/1Y3+oMQ/nWc4dKHvvcgSG87bM/Q/94BmuXOQHJlpX1uGSe2enO+rDbg2cGpZzTFkPfcAqHhpIYGM+gsz6ClY1OFm88nXMnskaClpvBi4XskkycCZrKBTOGKSn0vl93Z6JknYQJsrLuFE1vD54u0Zx2iqZ4yjR1iWa0uGR+WTw05X0uW9fsBngzXXs5ZiJmKlvAZetaSt52/soG+AT4xcHhsiWaAHD7hSvgE8Hr/RO47x2bEA8XSzRHU87evrZEGC2xUEmJ5ng6j0jQ2dHplM6Wz+B5SzRrdHA7ls6jbziF1jLPxcrGGgQsX8lQJdsSd8ro6YQBHhEREZG2blkM3/+9t+DcKgnqvLo7E9jYUbm+ykTEj8ZoEJef3YJL1zYDAO69cSM2tsfxtZ1OmV+hoPDp/96Dd3z+MQxNZPC37z4P77norFN2DZ11ERwYnEA6l8fhkRQ668PY3OF8r7/xjHMNbYkQupqjugcv52bwojoDtH9gAssbStd9rGl2grTpghmguN5hpum0k9ckjJX04FkI+61pBwh5p2dGgzZ84uzvMxnDcgHeBy5Z5WbA5tqf2Vwbcj/fFWc3l7xtfVsM77loJYDyPYkA0BoP46NXrsUnrl6Hbfo8mCFR+wcmkMkV0BYPuRlfs8rCGTxT7Et0h6xIMYMXD/uRzObdATBOBs9ypmgOp8o+F+9780r8xc2bSu6zfb6SJeunC5ZoEhEREXms0dkYOrXuvnIt0rkCLJ/g7qvW4tJ1TdjcmcAtb+rEH31zF/78e7/E4eEUvv7sIby9uw1/esOGU76qo7M+jG+/0If9/RNQyimzfOOKOjRGA/jSY87C87Z4GGtbavHtF/ownMzivOXOvrRIwMJ4Jo/jo2msay09IyaDN1OQZMpjEzN8TSZYyRUKePr1AbzUN+IGT021Qbcssvz7FvtGb+huR1dzVPfgOb/uL4tNDWo66yP4tXPb8fDTB+dcomn5BMviIYynczhXP0deH71yDR7Zfbhk+MpkH9y2quS2WW9ipn62JsLIFRRS2QJGkjkcHJrA4ETWzR6G/ZabuXOnaIZsxPRz3DechKUzfdGgjdF0FgcHJ9xdfl4rGmqwoqF0OJX/NJ2iyQCPiIiIiBZcz1n17utdzVF06azXrT2deLF3GH/7470AgLuvXIO7Lu2a9f7BueisiyBfUHhC99wtr4/Atny4ekMrvvJzHeAlwrj9whX4wUtH8IuDw27vW03AxjP7BzGayuHKSWsgzCTN6bJVwCwzeLqHbnA8i3u+/gLaE2F84BInCPrYleswpnvKyr+vuD184YDlPt8JHWCVK0sEgN+5rAu7Dg1jdXO07Ntncv3mNkQCU3cTAk5Z5I7f3jptT2I5UV2iuXOf04vXFg8jo/f0HR5J4S8feRmxkI136Wm1kYAFE5f6ROATJ0g0/Zwv9o6gJlAckvPUvgGksgVcuKphVtdj+U7PEs0FDfBEZDuAvwFgAfiCUupTk94eBPBlAG8E0A/gVqXUvoW8JiIiIiJaOgK2D/feuAlbuxqRyyu8/dz2BftcZhDIjud6ARQHpVy7yQnwRJxSRr/lw0O/eT7u/vfncfFqZ8jO9g3L8LoesHL+yvopH3t1c23JtMfJwgELtUHbnXBZTtDvTMK8/7/3AAAeuKPH7b2LR/wl/XuTbVvXjKbY1N5TE1C2TBPgrWiowfc+8pZpP+5MPr595p7OufbCmtLLl4+MYmN7HGuWRZHKOYN1/ur7L+NHLx/Dx7evdZ+HSMByg9rLz26G5QNEBJesaUJtyMbzB4fRpr/uaNBGKltAbciesqdxOn7Lh7Hc9EH1UrVgAZ6IWAA+B+CtAA4CeEpEdiilXvQ87H0ABpVSXSJyG4D7ANy6UNdEREREREvTdZvaFvxzmIDuyX0DuHh1Ixp1APKms+rRVBuEJeKWWdaG/Pj723vc9725p3PGVRgf2LYKR08Q4M0UoAFOGec/3t6D3uEkltdH3N602bh+cxuu3zz1OTTljPPdjbkYLJ/gbZvb0FIbxMe2r0XQtrCmuRZvaKzBo788io66MO7UvX0AsL4t7u6v29rV6E68jYf9uHPrSnzm0VfcrKAJlN+2uW3W02Q/uG3VaVmiKaZh8ZR/YJELAfyJUuoqffseAFBK3et5zCP6MY+LiA3gMIAmNcNF9fT0qJ07dy7INRMRERFRdfvaUwewsqkGPSvqSspAH3ridfSPZfC7ekH6qfbcgSEcHUlNKe9cDK8eHcWqpuiClL0ulkJBQQGzLvkcnsjizff9EF0tUXzjQ1vxJzt240uP7cM3PnRR2Z7B042IPK2U6in3toUs0WwHcMBz+yCA86d7jFIqJyLDABoAHF/A6yIiIiKiM9QtbyqfhXv3+SsW9PNWct1GV/PpPzjIN4dePsApaf30bd3uGoW3rm9BQamqWXsyk9NiyIqIvB/A+wFg+fLlFb4aIiIiIiJa6i4/u7ifz1vCWe0Wcg/eIQDeP5F06PvKPkaXaMbhDFspoZT6B6VUj1Kqp6lpfgsuiYiIiIiIqtVCBnhPAVgtIitFJADgNgA7Jj1mB4A79Os3AfjhTP13RERERERENL0FK9HUPXW/DeAROGsSvqiU2i0inwSwUym1A8ADAL4iIq8CGIATBBIREREREdFJWNAePKXUdwB8Z9J9/8fzegrAzQt5DURERERERGeKhSzRJCIiIiIiokXEAI+IiIiIiKhKMMAjIiIiIiKqEgzwiIiIiIiIqgQDPCIiIiIioirBAI+IiIiIiKhKMMAjIiIiIiKqEgzwiIiIiIiIqgQDPCIiIiIioirBAI+IiIiIiKhKMMAjIiIiIiKqEgzwiIiIiIiIqgQDPCIiIiIioirBAI+IiIiIiKhKiFKq0tcwJyJyDMDrlb6OMhoBHK/0RdAZi+ePKo1nkCqJ548qieePKmGFUqqp3BtOuwBvqRKRnUqpnkpfB52ZeP6o0ngGqZJ4/qiSeP5oqWGJJhERERERUZVggEdERERERFQlGOCdOv9Q6QugMxrPH1UazyBVEs8fVRLPHy0p7MEjIiIiIiKqEszgERERERERVQkGeKeAiGwXkZdF5FUR+USlr4eqj4h8UUSOisguz331IvIDEXlFv6zT94uIfEafx1+IyHmVu3KqBiLSKSI/EpEXRWS3iHxY388zSAtOREIi8qSIPK/P35/q+1eKyBP6nP2biAT0/UF9+1X99rMqef1UHUTEEpFnReS/9G2eP1qyGODNk4hYAD4H4GoA6wG8U0TWV/aqqAp9CcD2Sfd9AsCjSqnVAB7VtwHnLK7W/70fwOcX6RqpeuUAfFQptR7ABQDu0v/O8QzSYkgDuEwptRlAN4DtInIBgPsA3K+U6gIwCOB9+vHvAzCo779fP45ovj4M4CXPbZ4/WrIY4M3fFgCvKqVeU0plAPwrgBsqfE1UZZRS/wtgYNLdNwB4UL/+IIC3e+7/snL8HEBCRFoX50qpGiml+pRSz+jXR+H8ktMOnkFaBPocjembfv2fAnAZgIf1/ZPPnzmXDwO4XERkkS6XqpCIdAC4FsAX9G0Bzx8tYQzw5q8dwAHP7YP6PqKF1qKU6tOvHwbQol/nmaQFo8uNzgXwBHgGaZHo8rjnABwF8AMAewEMKaVy+iHeM+aeP/32YQANi3vFVGU+DeDjAAr6dgN4/mgJY4BHVAWUMw6XI3FpQYlIFMB/APiIUmrE+zaeQVpISqm8UqobQAecypl1Fb4kOkOIyHUAjiqlnq70tRDNFgO8+TsEoNNzu0PfR7TQjpiyN/3yqL6fZ5JOORHxwwnuHlJKfV3fzTNIi0opNQTgRwAuhFP6a+s3ec+Ye/702+MA+hf5Uql6bAVwvYjsg9OGcxmAvwHPHy1hDPDm7ykAq/U0pQCA2wDsqPA10ZlhB4A79Ot3APiW5/7f0JMMLwAw7CmjI5oz3T/yAICXlFJ/7XkTzyAtOBFpEpGEfj0M4K1w+kB/BOAm/bDJ58+cy5sA/FBx6S+dJKXUPUqpDqXUWXB+x/uhUvkt164AAALdSURBVOrd4PmjJYyLzk8BEbkGTn22BeCLSqn/V+FLoiojIl8FsA1AI4AjAP4YwDcBfA3AcgCvA7hFKTWgfxn/LJypmxMA7lRK7azEdVN1EJE3A/gJgBdQ7EH5Azh9eDyDtKBEZBOcoRUWnD9Mf00p9UkReQOcjEo9gGcB/LpSKi0iIQBfgdMrOgDgNqXUa5W5eqomIrINwN1Kqet4/mgpY4BHRERERERUJViiSUREREREVCUY4BEREREREVUJBnhERERERERVggEeERERERFRlWCAR0REREREVCUY4BER0RlLRPIi8pyIPC8iz4jIRSd4fEJEPjSLj/tjEek5dVdKREQ0OwzwiIjoTJZUSnUrpTYDuAfAvSd4fALACQM8IiKiSmGAR0RE5IgBGAQAEYmKyKM6q/eCiNygH/MpAKt01u8v9GN/Xz/meRH5lOfj3SwiT4rIHhG5eHG/FCIiOlPZlb4AIiKiCgqLyHMAQgBaAVym708B+DWl1IiINAL4uYjsAPAJABuUUt0AICJXA7gBwPlKqQkRqfd8bFsptUVErgHwxwCuWKSviYiIzmAM8IiI6EyW9ARrFwL4sohsACAA/kxE3gKgAKAdQEuZ978CwD8ppSYAQCk14Hnb1/XLpwGctTCXT0REVIoBHhEREQCl1OM6W9cE4Br98o1KqayI7IOT5ZuLtH6ZB/9/S0REi4Q9eERERABEZB0AC0A/gDiAozq4uxTACv2wUQC1nnf7AYA7RSSiP4a3RJOIiGjR8S+KRER0JjM9eIBTlnmHUiovIg8B+E8ReQHATgC/BAClVL+I/ExEdgH4rlLqYyLSDWCniGQAfAfAH1Tg6yAiIgIAiFKq0tdAREREREREpwBLNImIiIiIiKoEAzwiIiIiIqIqwQCPiIiIiIioSjDAIyIiIiIiqhIM8IiIiIiIiKoEAzwiIiIiIqIqwQCPiIiIiIioSjDAIyIiIiIiqhL/H/z+0usOkecrAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" ] - }, - { - "cell_type": "code", - "metadata": { - "id": "K3dXaA9DhQ6W", - "colab_type": "code", - "colab": {} - }, - "source": [ - "" - ], - "execution_count": 0, - "outputs": [] + }, + "metadata": { + "needs_background": "light", + "tags": [] + }, + "output_type": "display_data" } - ] -} \ No newline at end of file + ], + "source": [ + "# BERT fine-tuning parameters\n", + "param_optimizer = list(model.named_parameters())\n", + "no_decay = ['bias', 'gamma', 'beta']\n", + "optimizer_grouped_parameters = [\n", + " {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n", + " 'weight_decay_rate': 0.01},\n", + " {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n", + " 'weight_decay_rate': 0.0}\n", + "]\n", + "\n", + "optimizer = BertAdam(optimizer_grouped_parameters,\n", + " lr=2e-5,\n", + " warmup=.1)\n", + "\n", + "# Function to calculate the accuracy of our predictions vs labels\n", + "def flat_accuracy(preds, labels):\n", + " pred_flat = np.argmax(preds, axis=1).flatten()\n", + " labels_flat = labels.flatten()\n", + " return np.sum(pred_flat == labels_flat) / len(labels_flat)\n", + "\n", + "torch.cuda.empty_cache() \n", + "# Store our loss and accuracy for plotting\n", + "train_loss_set = []\n", + "# Number of training epochs \n", + "epochs = 4\n", + "\n", + "# BERT training loop\n", + "for _ in trange(epochs, desc=\"Epoch\"): \n", + " \n", + " ## TRAINING\n", + "\n", + " # Set our model to training mode\n", + " model.train() \n", + " # Tracking variables\n", + " tr_loss = 0\n", + " nb_tr_examples, nb_tr_steps = 0, 0\n", + " # Train the data for one epoch\n", + " for step, batch in enumerate(train_dataloader):\n", + " # Add batch to GPU\n", + " batch = tuple(t.to(device) for t in batch)\n", + " # Unpack the inputs from our dataloader\n", + " b_input_ids, b_input_mask, b_labels = batch\n", + " # Clear out the gradients (by default they accumulate)\n", + " optimizer.zero_grad()\n", + " # Forward pass\n", + " loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n", + " train_loss_set.append(loss.item()) \n", + " # Backward pass\n", + " loss.backward()\n", + " # Update parameters and take a step using the computed gradient\n", + " optimizer.step()\n", + " # Update tracking variables\n", + " tr_loss += loss.item()\n", + " nb_tr_examples += b_input_ids.size(0)\n", + " nb_tr_steps += 1\n", + " print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n", + "\n", + " ## VALIDATION\n", + "\n", + " # Put model in evaluation mode\n", + " model.eval()\n", + " # Tracking variables \n", + " eval_loss, eval_accuracy = 0, 0\n", + " nb_eval_steps, nb_eval_examples = 0, 0\n", + " # Evaluate data for one epoch\n", + " for batch in validation_dataloader:\n", + " # Add batch to GPU\n", + " batch = tuple(t.to(device) for t in batch)\n", + " # Unpack the inputs from our dataloader\n", + " b_input_ids, b_input_mask, b_labels = batch\n", + " # Telling the model not to compute or store gradients, saving memory and speeding up validation\n", + " with torch.no_grad():\n", + " # Forward pass, calculate logit predictions\n", + " logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask) \n", + " # Move logits and labels to CPU\n", + " logits = logits.detach().cpu().numpy()\n", + " label_ids = b_labels.to('cpu').numpy()\n", + " tmp_eval_accuracy = flat_accuracy(logits, label_ids) \n", + " eval_accuracy += tmp_eval_accuracy\n", + " nb_eval_steps += 1\n", + " print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n", + "\n", + "# plot training performance\n", + "plt.figure(figsize=(15,8))\n", + "plt.title(\"Training loss\")\n", + "plt.xlabel(\"Batch\")\n", + "plt.ylabel(\"Loss\")\n", + "plt.plot(train_loss_set)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "78yvfEVHoncI" + }, + "source": [ + "We achieved a validation accuracy of 73% in the last epoch. This is just a demonstration, we might be able to achieve higher accuracies by performing some pre-processing steps and hyper parameter tuning." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "K3dXaA9DhQ6W" + }, + "outputs": [], + "source": [] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "collapsed_sections": [], + "name": "03_FinBERT.ipynb", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.13" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/Ch11/01_AutoML_Demo.ipynb b/Ch11/01_AutoML_Demo.ipynb index 6f36e25..c5aae5b 100644 --- a/Ch11/01_AutoML_Demo.ipynb +++ b/Ch11/01_AutoML_Demo.ipynb @@ -3,46 +3,163 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "iXFYlwdPNans" }, "source": [ - "In this notebook we will demonstrate Auto ML on the mnist digits dataset. More information about autosklearn and autokeras can we found in the official documents [here](https://automl.github.io/auto-sklearn/master) and [here](https://github.com/keras-team/autokeras) respectively.
\n", + "In this notebook we will demonstrate Auto ML on the mnist digits dataset. More information about autosklearn and autokeras can be found in the official documents [here](https://automl.github.io/auto-sklearn/master) and [here](https://github.com/keras-team/autokeras) respectively.
\n", "Note: Auto-sklearn is not available on windows or on MAC OSX." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "P2zerVlHXoIb", - "outputId": "6b36598e-98f1-4abe-cbe7-2edd9381c172" + "outputId": "43bbf936-34bd-403e-8778-9a4cbe147fa9" }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Reading package lists... Done\n", + "Building dependency tree \n", + "Reading state information... Done\n", + "build-essential is already the newest version (12.4ubuntu1).\n", + "swig is already the newest version (3.0.12-1).\n", + "0 upgraded, 0 newly installed, 0 to remove and 39 not upgraded.\n", + " % Total % Received % Xferd Average Speed Time Time Time Current\n", + " Dload Upload Total Spent Left Speed\n", + "100 209 100 209 0 0 6531 0 --:--:-- --:--:-- --:--:-- 6531\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (57.0.0)\n", + "Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.7/dist-packages (1.19.5)\n", + "Requirement already satisfied: scipy>=0.14.1 in /usr/local/lib/python3.7/dist-packages (1.4.1)\n", + "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from scipy>=0.14.1) (1.19.5)\n", + "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (1.0.1)\n", + "Requirement already satisfied: scikit-learn<0.25.0,>=0.24.0 in /usr/local/lib/python3.7/dist-packages (0.24.0)\n", + "Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from scikit-learn<0.25.0,>=0.24.0) (1.4.1)\n", + "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from scikit-learn<0.25.0,>=0.24.0) (1.19.5)\n", + "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn<0.25.0,>=0.24.0) (1.0.1)\n", + "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn<0.25.0,>=0.24.0) (2.1.0)\n", + "Requirement already satisfied: dask in /usr/local/lib/python3.7/dist-packages (2021.6.0)\n", + "Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from dask) (1.6.0)\n", + "Requirement already satisfied: fsspec>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from dask) (2021.6.0)\n", + "Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.7/dist-packages (from dask) (0.11.1)\n", + "Requirement already satisfied: partd>=0.3.10 in /usr/local/lib/python3.7/dist-packages (from dask) (1.2.0)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from dask) (3.13)\n", + "Requirement already satisfied: locket in /usr/local/lib/python3.7/dist-packages (from partd>=0.3.10->dask) (0.2.1)\n", + "Requirement already satisfied: distributed>=2.2.0 in /usr/local/lib/python3.7/dist-packages (2021.6.0)\n", + "Requirement already satisfied: psutil>=5.0 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (5.4.8)\n", + "Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (2.4.0)\n", + "Requirement already satisfied: tblib>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (1.7.0)\n", + "Requirement already satisfied: dask==2021.06.0 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (2021.6.0)\n", + "Requirement already satisfied: tornado>=5; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (5.1.1)\n", + "Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (1.0.2)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (3.13)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (57.0.0)\n", + "Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (7.1.2)\n", + "Requirement already satisfied: zict>=0.1.3 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (2.0.0)\n", + "Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (0.11.1)\n", + "Requirement already satisfied: cloudpickle>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0) (1.6.0)\n", + "Requirement already satisfied: partd>=0.3.10 in /usr/local/lib/python3.7/dist-packages (from dask==2021.06.0->distributed>=2.2.0) (1.2.0)\n", + "Requirement already satisfied: fsspec>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from dask==2021.06.0->distributed>=2.2.0) (2021.6.0)\n", + "Requirement already satisfied: heapdict in /usr/local/lib/python3.7/dist-packages (from zict>=0.1.3->distributed>=2.2.0) (1.0.1)\n", + "Requirement already satisfied: locket in /usr/local/lib/python3.7/dist-packages (from partd>=0.3.10->dask==2021.06.0->distributed>=2.2.0) (0.2.1)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (3.13)\n", + "Requirement already satisfied: pandas>=1.0 in /usr/local/lib/python3.7/dist-packages (1.1.5)\n", + "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0) (2.8.1)\n", + "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0) (2018.9)\n", + "Requirement already satisfied: numpy>=1.15.4 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0) (1.19.5)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas>=1.0) (1.15.0)\n", + "Requirement already satisfied: liac-arff in /usr/local/lib/python3.7/dist-packages (2.5.0)\n", + "Requirement already satisfied: ConfigSpace<0.5,>=0.4.14 in /usr/local/lib/python3.7/dist-packages (0.4.19)\n", + "Requirement already satisfied: pyparsing in /usr/local/lib/python3.7/dist-packages (from ConfigSpace<0.5,>=0.4.14) (2.4.7)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from ConfigSpace<0.5,>=0.4.14) (1.19.5)\n", + "Requirement already satisfied: cython in /usr/local/lib/python3.7/dist-packages (from ConfigSpace<0.5,>=0.4.14) (0.29.23)\n", + "Requirement already satisfied: pynisher>=0.6.3 in /usr/local/lib/python3.7/dist-packages (0.6.4)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from pynisher>=0.6.3) (57.0.0)\n", + "Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from pynisher>=0.6.3) (5.4.8)\n", + "Requirement already satisfied: pyrfr<0.9,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (0.8.2)\n", + "Requirement already satisfied: smac<0.14,>=0.13.1 in /usr/local/lib/python3.7/dist-packages (0.13.1)\n", + "Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (5.4.8)\n", + "Requirement already satisfied: pynisher>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (0.6.4)\n", + "Requirement already satisfied: pyrfr>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (0.8.2)\n", + "Requirement already satisfied: dask in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (2021.6.0)\n", + "Requirement already satisfied: lazy-import in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (0.2.2)\n", + "Requirement already satisfied: scikit-learn>=0.22.0 in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (0.24.0)\n", + "Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (1.4.1)\n", + "Requirement already satisfied: ConfigSpace<0.5,>=0.4.14 in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (0.4.19)\n", + "Requirement already satisfied: distributed in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (2021.6.0)\n", + "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (1.0.1)\n", + "Requirement already satisfied: numpy>=1.7.1 in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1) (1.19.5)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from pynisher>=0.4.1->smac<0.14,>=0.13.1) (57.0.0)\n", + "Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.7/dist-packages (from dask->smac<0.14,>=0.13.1) (0.11.1)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from dask->smac<0.14,>=0.13.1) (3.13)\n", + "Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from dask->smac<0.14,>=0.13.1) (1.6.0)\n", + "Requirement already satisfied: fsspec>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from dask->smac<0.14,>=0.13.1) (2021.6.0)\n", + "Requirement already satisfied: partd>=0.3.10 in /usr/local/lib/python3.7/dist-packages (from dask->smac<0.14,>=0.13.1) (1.2.0)\n", + "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from lazy-import->smac<0.14,>=0.13.1) (1.15.0)\n", + "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.22.0->smac<0.14,>=0.13.1) (2.1.0)\n", + "Requirement already satisfied: cython in /usr/local/lib/python3.7/dist-packages (from ConfigSpace<0.5,>=0.4.14->smac<0.14,>=0.13.1) (0.29.23)\n", + "Requirement already satisfied: pyparsing in /usr/local/lib/python3.7/dist-packages (from ConfigSpace<0.5,>=0.4.14->smac<0.14,>=0.13.1) (2.4.7)\n", + "Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /usr/local/lib/python3.7/dist-packages (from distributed->smac<0.14,>=0.13.1) (2.4.0)\n", + "Requirement already satisfied: tornado>=5; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from distributed->smac<0.14,>=0.13.1) (5.1.1)\n", + "Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.7/dist-packages (from distributed->smac<0.14,>=0.13.1) (7.1.2)\n", + "Requirement already satisfied: zict>=0.1.3 in /usr/local/lib/python3.7/dist-packages (from distributed->smac<0.14,>=0.13.1) (2.0.0)\n", + "Requirement already satisfied: tblib>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from distributed->smac<0.14,>=0.13.1) (1.7.0)\n", + "Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from distributed->smac<0.14,>=0.13.1) (1.0.2)\n", + "Requirement already satisfied: locket in /usr/local/lib/python3.7/dist-packages (from partd>=0.3.10->dask->smac<0.14,>=0.13.1) (0.2.1)\n", + "Requirement already satisfied: heapdict in /usr/local/lib/python3.7/dist-packages (from zict>=0.1.3->distributed->smac<0.14,>=0.13.1) (1.0.1)\n", + "Requirement already satisfied: auto-sklearn in /usr/local/lib/python3.7/dist-packages (0.12.6)\n", + "Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (1.19.5)\n", + "Requirement already satisfied: pandas>=1.0 in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (1.1.5)\n", + "Requirement already satisfied: dask in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (2021.6.0)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (3.13)\n", + "Requirement already satisfied: scikit-learn<0.25.0,>=0.24.0 in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (0.24.0)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (57.0.0)\n", + "Requirement already satisfied: pynisher>=0.6.3 in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (0.6.4)\n", + "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (1.0.1)\n", + "Requirement already satisfied: smac<0.14,>=0.13.1 in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (0.13.1)\n", + "Requirement already satisfied: ConfigSpace<0.5,>=0.4.14 in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (0.4.19)\n", + "Requirement already satisfied: distributed>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (2021.6.0)\n", + "Requirement already satisfied: scipy>=0.14.1 in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (1.4.1)\n", + "Requirement already satisfied: pyrfr<0.9,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (0.8.2)\n", + "Requirement already satisfied: liac-arff in /usr/local/lib/python3.7/dist-packages (from auto-sklearn) (2.5.0)\n", + "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0->auto-sklearn) (2.8.1)\n", + "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0->auto-sklearn) (2018.9)\n", + "Requirement already satisfied: fsspec>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from dask->auto-sklearn) (2021.6.0)\n", + "Requirement already satisfied: partd>=0.3.10 in /usr/local/lib/python3.7/dist-packages (from dask->auto-sklearn) (1.2.0)\n", + "Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.7/dist-packages (from dask->auto-sklearn) (0.11.1)\n", + "Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from dask->auto-sklearn) (1.6.0)\n", + "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn<0.25.0,>=0.24.0->auto-sklearn) (2.1.0)\n", + "Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from pynisher>=0.6.3->auto-sklearn) (5.4.8)\n", + "Requirement already satisfied: lazy-import in /usr/local/lib/python3.7/dist-packages (from smac<0.14,>=0.13.1->auto-sklearn) (0.2.2)\n", + "Requirement already satisfied: cython in /usr/local/lib/python3.7/dist-packages (from ConfigSpace<0.5,>=0.4.14->auto-sklearn) (0.29.23)\n", + "Requirement already satisfied: pyparsing in /usr/local/lib/python3.7/dist-packages (from ConfigSpace<0.5,>=0.4.14->auto-sklearn) (2.4.7)\n", + "Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0->auto-sklearn) (2.4.0)\n", + "Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0->auto-sklearn) (7.1.2)\n", + "Requirement already satisfied: zict>=0.1.3 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0->auto-sklearn) (2.0.0)\n", + "Requirement already satisfied: tornado>=5; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0->auto-sklearn) (5.1.1)\n", + "Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0->auto-sklearn) (1.0.2)\n", + "Requirement already satisfied: tblib>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from distributed>=2.2.0->auto-sklearn) (1.7.0)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas>=1.0->auto-sklearn) (1.15.0)\n", + "Requirement already satisfied: locket in /usr/local/lib/python3.7/dist-packages (from partd>=0.3.10->dask->auto-sklearn) (0.2.1)\n", + "Requirement already satisfied: heapdict in /usr/local/lib/python3.7/dist-packages (from zict>=0.1.3->distributed>=2.2.0->auto-sklearn) (1.0.1)\n" + ] + } + ], "source": [ "!sudo apt-get install build-essential swig\n", "!curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | xargs -n 1 -L 1 pip install\n", - "!pip install auto-sklearn\n", - "\n", - "for _ in range(3):\n", - " try:\n", - " import autosklearn.classification\n", - " break\n", - " except:\n", - " pass\n", - "else:\n", - " raise ImportError(\"failed to import from autosklearn\")" + "!pip install auto-sklearn" ] }, { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "YoYPpFPnN7yc" }, "source": [ @@ -51,10 +168,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 2, "metadata": { - "colab": {}, - "colab_type": "code", "id": "-Ch8b0O3ywBG" }, "outputs": [], @@ -72,15 +187,13 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "Qv-fKz1V3Pvx", - "outputId": "0016da10-d091-4c1f-877f-3c3842d3e369" + "outputId": "60f3d395-0a1f-432a-c99c-0244021ff76c" }, "outputs": [ { @@ -110,7 +223,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "eM-USO8VYiOr" }, "source": [ @@ -120,15 +232,13 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "RJgRfTBmXxfr", - "outputId": "e6fb966c-09a6-4f99-d695-a8ea49688237" + "outputId": "b1112f51-56b0-4a08-9d5f-32a764942ea1" }, "outputs": [ { @@ -136,47 +246,7 @@ "output_type": "stream", "text": [ "Model\n", - " [(0.080000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 258.9142997145647, 'classifier:libsvm_svc:gamma': 0.008836682811147231, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.00017468203025368254, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7800242369785717, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.25089878093468937, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.903450435738307, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0002126471280988806, 'classifier:libsvm_svc:coef0': 0.5003202726941338, 'classifier:libsvm_svc:degree': 3},\n", - "dataset_properties={\n", - " 'task': 2,\n", - " 'sparse': False,\n", - " 'multilabel': False,\n", - " 'multiclass': True,\n", - " 'target_type': 'classification',\n", - " 'signed': False})),\n", - "(0.060000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 179.5810613582497, 'classifier:libsvm_svc:gamma': 0.008836682811147231, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0006392825862463574, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.010911047530006813, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7168585962550419, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.25, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.012213572770343, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0002126471280988806, 'classifier:libsvm_svc:coef0': 0.4922337405686199, 'classifier:libsvm_svc:degree': 4},\n", - "dataset_properties={\n", - " 'task': 2,\n", - " 'sparse': False,\n", - " 'multilabel': False,\n", - " 'multiclass': True,\n", - " 'target_type': 'classification',\n", - " 'signed': False})),\n", - "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 39.947366669501925, 'classifier:libsvm_svc:gamma': 0.008836682811147231, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0007012962659233003, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.00239516876546622, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.9734849446794978, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0002126471280988806, 'classifier:libsvm_svc:coef0': 0.4504739926690646, 'classifier:libsvm_svc:degree': 3},\n", - "dataset_properties={\n", - " 'task': 2,\n", - " 'sparse': False,\n", - " 'multilabel': False,\n", - " 'multiclass': True,\n", - " 'target_type': 'classification',\n", - " 'signed': False})),\n", - "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 25.519264309557933, 'classifier:libsvm_svc:gamma': 0.008851468557757787, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0001120413413778449, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7541218755872775, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.22188350305005386, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.4754586245917936, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00020444706315893564, 'classifier:libsvm_svc:coef0': 0.4880519484063368, 'classifier:libsvm_svc:degree': 3},\n", - "dataset_properties={\n", - " 'task': 2,\n", - " 'sparse': False,\n", - " 'multilabel': False,\n", - " 'multiclass': True,\n", - " 'target_type': 'classification',\n", - " 'signed': False})),\n", - "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'none', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 168.3535117139133, 'classifier:libsvm_svc:gamma': 0.008378099742924994, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0006529870495855917, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.01055532524571016, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.9196503388135624, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00023294010806870216, 'classifier:libsvm_svc:coef0': 0.49212957683863046, 'classifier:libsvm_svc:degree': 3},\n", - "dataset_properties={\n", - " 'task': 2,\n", - " 'sparse': False,\n", - " 'multilabel': False,\n", - " 'multiclass': True,\n", - " 'target_type': 'classification',\n", - " 'signed': False})),\n", - "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 0.5579746801991264, 'classifier:libsvm_svc:gamma': 0.029748789483583407, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0002839137063809134, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.8182544628000112, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00025630000569846123, 'classifier:libsvm_svc:coef0': 0.5083445743390529, 'classifier:libsvm_svc:degree': 3},\n", + " [(0.060000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 24427.51442319513, 'classifier:libsvm_svc:gamma': 0.0003734294061388043, 'classifier:libsvm_svc:kernel': 'sigmoid', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.020946604527726176, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'True', 'classifier:libsvm_svc:coef0': -0.10707429456459294},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -184,7 +254,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 177.149886658746, 'classifier:libsvm_svc:gamma': 0.008836682811147231, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0007012962659233003, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7112420757119524, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.2308431158905038, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.863759655230442, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0002126471280988806, 'classifier:libsvm_svc:coef0': 0.5806835435976878, 'classifier:libsvm_svc:degree': 3},\n", + "(0.060000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 10348.145321615948, 'classifier:libsvm_svc:gamma': 0.00028793595031627306, 'classifier:libsvm_svc:kernel': 'sigmoid', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.06385554987109476, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True', 'classifier:libsvm_svc:coef0': 0.0},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -192,7 +262,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 1.4452196732269074, 'classifier:libsvm_svc:gamma': 0.0332315022897029, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0007652960853112059, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.859008848776207, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00025348965052995166, 'classifier:libsvm_svc:coef0': 0.5435894165902586, 'classifier:libsvm_svc:degree': 3},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 21414.598775704668, 'classifier:libsvm_svc:gamma': 0.00023021107501257278, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.04421516395928401, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.005529432906989406, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -200,7 +270,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 147.23346600674694, 'classifier:libsvm_svc:gamma': 0.010403007349729874, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0007012962659233003, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.4588164343934653, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00020878937270087631, 'classifier:libsvm_svc:coef0': 0.44124771200090174, 'classifier:libsvm_svc:degree': 3},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 23047.23607871643, 'classifier:libsvm_svc:gamma': 0.00029990641773151107, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.030822435566314237, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.0013630791240315208, 'feature_preprocessor:polynomial:degree': 2, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -208,7 +278,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'none', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 179.5810613582497, 'classifier:libsvm_svc:gamma': 0.008836682811147231, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0007012962659233003, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.863759655230442, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0002126471280988806, 'classifier:libsvm_svc:coef0': 0.4781617760830539, 'classifier:libsvm_svc:degree': 3},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 23742.684467409774, 'classifier:libsvm_svc:gamma': 0.00032231839019304935, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.06343418459512941, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.0010283829230508007, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'False'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -216,7 +286,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'extra_trees_preproc_for_classification', 'classifier:libsvm_svc:C': 32106.27648314644, 'classifier:libsvm_svc:gamma': 0.033170457043270415, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.005164979868372482, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.01710064098545451, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.75, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.2582579368716088, 'feature_preprocessor:extra_trees_preproc_for_classification:bootstrap': 'True', 'feature_preprocessor:extra_trees_preproc_for_classification:criterion': 'gini', 'feature_preprocessor:extra_trees_preproc_for_classification:max_depth': 'None', 'feature_preprocessor:extra_trees_preproc_for_classification:max_features': 0.2865376682948705, 'feature_preprocessor:extra_trees_preproc_for_classification:max_leaf_nodes': 'None', 'feature_preprocessor:extra_trees_preproc_for_classification:min_impurity_decrease': 0.0, 'feature_preprocessor:extra_trees_preproc_for_classification:min_samples_leaf': 18, 'feature_preprocessor:extra_trees_preproc_for_classification:min_samples_split': 7, 'feature_preprocessor:extra_trees_preproc_for_classification:min_weight_fraction_leaf': 0.0, 'feature_preprocessor:extra_trees_preproc_for_classification:n_estimators': 100, 'classifier:libsvm_svc:coef0': 0.449251416716546, 'classifier:libsvm_svc:degree': 5},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 12478.680306970935, 'classifier:libsvm_svc:gamma': 0.0002551322530468197, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.06325577217516412, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'False'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -224,7 +294,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 511.99967909382303, 'classifier:libsvm_svc:gamma': 0.008836682811147231, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.00029959592509141053, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.010000000000000004, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7876788058409236, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.2300128832770751, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.903450435738307, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0002090759496769351, 'classifier:libsvm_svc:coef0': 0.49946222314586475, 'classifier:libsvm_svc:degree': 3},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 11645.510084896576, 'classifier:libsvm_svc:gamma': 0.00031550397242672895, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.05089122555545417, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'False'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -232,7 +302,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'standardize', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 266.9987051961645, 'classifier:libsvm_svc:gamma': 0.008787315435267475, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0006982719240031436, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.5214160806367936, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00021580405139699842, 'classifier:libsvm_svc:coef0': 0.5058119593663437, 'classifier:libsvm_svc:degree': 3},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 30991.44044952807, 'classifier:libsvm_svc:gamma': 0.00041386787620574744, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.02050752821628261, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'False'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -240,7 +310,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 3.1443945568790665, 'classifier:libsvm_svc:gamma': 0.020996921706550105, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.003997214921562833, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.80889764115218, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00025541299512113437, 'classifier:libsvm_svc:coef0': 0.36216008149751433, 'classifier:libsvm_svc:degree': 3},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 10025.14161656712, 'classifier:libsvm_svc:gamma': 0.0002992862526435906, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.05475519811331643, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -248,7 +318,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'lda', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'nystroem_sampler', 'classifier:lda:n_components': 76, 'classifier:lda:shrinkage': 'None', 'classifier:lda:tol': 0.09865556043445883, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.005823759089880865, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7978120173580654, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.2811219694282669, 'feature_preprocessor:nystroem_sampler:kernel': 'poly', 'feature_preprocessor:nystroem_sampler:n_components': 8610, 'feature_preprocessor:nystroem_sampler:coef0': 0.17624345903922523, 'feature_preprocessor:nystroem_sampler:degree': 2, 'feature_preprocessor:nystroem_sampler:gamma': 0.0009546886303843381},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 10346.173155512957, 'classifier:libsvm_svc:gamma': 0.00037735418714701305, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.03710317494678222, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'False'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -256,7 +326,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 31.018167908606582, 'classifier:libsvm_svc:gamma': 0.015679873358514684, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0007885174437878954, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7560912719261723, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.22829326936116728, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.836723352423671, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0002227280481237857, 'classifier:libsvm_svc:coef0': 0.4519923639377479, 'classifier:libsvm_svc:degree': 2},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 12006.41352151287, 'classifier:libsvm_svc:gamma': 0.0001420039537922682, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.03993387612134234, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -264,7 +334,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 2.391940639970641, 'classifier:libsvm_svc:gamma': 0.018508563213452928, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0007008235445707821, 'feature_preprocessor:liblinear_svc_preprocessor:C': 1.8736809179114424, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00040406509918139026, 'classifier:libsvm_svc:coef0': 0.44557340202535345, 'classifier:libsvm_svc:degree': 2},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 18170.395434601698, 'classifier:libsvm_svc:gamma': 0.0002873830740440994, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.03375358572065382, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -272,7 +342,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 14.17287565769628, 'classifier:libsvm_svc:gamma': 0.02082659124595437, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0008682073870124408, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.769930826833681, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00025350277545829644, 'classifier:libsvm_svc:coef0': 0.5118955719662039, 'classifier:libsvm_svc:degree': 4},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 16710.866392278756, 'classifier:libsvm_svc:gamma': 0.00029230977584756017, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.02766872102174871, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'False'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -280,7 +350,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 275.7847947237385, 'classifier:libsvm_svc:gamma': 0.010403007349729874, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.002022131781757993, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.010000000000000004, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.4588164343934653, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00020878937270087631, 'classifier:libsvm_svc:coef0': 0.07183585535219206, 'classifier:libsvm_svc:degree': 3},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 9226.932738982554, 'classifier:libsvm_svc:gamma': 0.0001510547303173578, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.04834400212815847, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.0035108194478721665, 'feature_preprocessor:polynomial:degree': 2, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -288,7 +358,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'quantile_transformer', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 131.88959548011042, 'classifier:libsvm_svc:gamma': 0.00908832698163621, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0004285539002925058, 'data_preprocessing:numerical_transformer:rescaling:quantile_transformer:n_quantiles': 1002, 'data_preprocessing:numerical_transformer:rescaling:quantile_transformer:output_distribution': 'normal', 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.9170208578180024, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00022128884750616953, 'classifier:libsvm_svc:coef0': 0.2061271059985359, 'classifier:libsvm_svc:degree': 4},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'none', 'feature_preprocessor:__choice__': 'no_preprocessing', 'classifier:libsvm_svc:C': 4426.081783249642, 'classifier:libsvm_svc:gamma': 0.0002618418414109557, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.07291817207847671, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.0124088264710248},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -296,7 +366,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'extra_trees_preproc_for_classification', 'classifier:libsvm_svc:C': 1657.961664168718, 'classifier:libsvm_svc:gamma': 0.028381746836309094, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.00213656823372383, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.010000000000000004, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.75, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.25816433437943537, 'feature_preprocessor:extra_trees_preproc_for_classification:bootstrap': 'True', 'feature_preprocessor:extra_trees_preproc_for_classification:criterion': 'gini', 'feature_preprocessor:extra_trees_preproc_for_classification:max_depth': 'None', 'feature_preprocessor:extra_trees_preproc_for_classification:max_features': 0.6722294607333421, 'feature_preprocessor:extra_trees_preproc_for_classification:max_leaf_nodes': 'None', 'feature_preprocessor:extra_trees_preproc_for_classification:min_impurity_decrease': 0.0, 'feature_preprocessor:extra_trees_preproc_for_classification:min_samples_leaf': 11, 'feature_preprocessor:extra_trees_preproc_for_classification:min_samples_split': 6, 'feature_preprocessor:extra_trees_preproc_for_classification:min_weight_fraction_leaf': 0.0, 'feature_preprocessor:extra_trees_preproc_for_classification:n_estimators': 100, 'classifier:libsvm_svc:coef0': 0.44894402381246756, 'classifier:libsvm_svc:degree': 5},\n", + "(0.040000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'no_preprocessing', 'classifier:libsvm_svc:C': 9730.142237419986, 'classifier:libsvm_svc:gamma': 0.7497678185408562, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0009086517364883961, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.007256009345525524},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -304,7 +374,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'pca', 'classifier:libsvm_svc:C': 1.463720504945052, 'classifier:libsvm_svc:gamma': 0.009559157980168218, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0003634977031188347, 'feature_preprocessor:pca:keep_variance': 0.9888654707281754, 'feature_preprocessor:pca:whiten': 'False', 'classifier:libsvm_svc:coef0': 0.4572988730246441, 'classifier:libsvm_svc:degree': 4},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 19729.156463287323, 'classifier:libsvm_svc:gamma': 0.00024014080141349147, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.03438117253566002, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.004216928716385872, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -312,7 +382,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'pca', 'classifier:libsvm_svc:C': 4.307686690368154, 'classifier:libsvm_svc:gamma': 0.010222676758767703, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0003240139317364383, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.010000000000000004, 'feature_preprocessor:pca:keep_variance': 0.9890854982851038, 'feature_preprocessor:pca:whiten': 'False', 'classifier:libsvm_svc:coef0': 0.48806662877288076, 'classifier:libsvm_svc:degree': 3},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 23047.23607871643, 'classifier:libsvm_svc:gamma': 0.00029990641773151107, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.047131318165309015, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.0031279222016672614, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -320,7 +390,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 233.4421847143784, 'classifier:libsvm_svc:gamma': 0.0043753811702647305, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0008369469383461333, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7005856652474891, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.2705585015631023, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.6355754861482006, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00020878937270087631, 'classifier:libsvm_svc:coef0': 0.49459099187381295, 'classifier:libsvm_svc:degree': 4},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 14668.421225885211, 'classifier:libsvm_svc:gamma': 0.00022942654247324606, 'classifier:libsvm_svc:kernel': 'sigmoid', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.04680842194545637, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'False', 'classifier:libsvm_svc:coef0': -0.38626941120932423},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -328,7 +398,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 170.7190663132324, 'classifier:libsvm_svc:gamma': 0.008836682811147231, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0001555630285051608, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7525461592866844, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.21481323374945413, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.903450435738307, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0002090759496769351, 'classifier:libsvm_svc:coef0': 0.49946222314586475, 'classifier:libsvm_svc:degree': 4},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 27320.653183313574, 'classifier:libsvm_svc:gamma': 0.0003255372779127976, 'classifier:libsvm_svc:kernel': 'sigmoid', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.06321975285919286, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'False', 'classifier:libsvm_svc:coef0': 0.1358044316020992},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -336,7 +406,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'standardize', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 57.91923577771208, 'classifier:libsvm_svc:gamma': 0.010818608352224311, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0006663402398851591, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.009097996867637787, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.863759655230442, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00021927949546088335, 'classifier:libsvm_svc:coef0': 0.05730572601032491, 'classifier:libsvm_svc:degree': 3},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 27165.336200638478, 'classifier:libsvm_svc:gamma': 0.0003579437150864718, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.023448181943886124, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'False'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -344,7 +414,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'standardize', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 173.82144092828102, 'classifier:libsvm_svc:gamma': 0.008851468557757787, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0001120413413778449, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.010000000000000004, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.4754586245917936, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00020444706315893564, 'classifier:libsvm_svc:coef0': 0.01896428310362941, 'classifier:libsvm_svc:degree': 3},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 6655.219975032081, 'classifier:libsvm_svc:gamma': 0.0006392164844545438, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.06506191873299852, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -352,7 +422,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'standardize', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 33.470207362664, 'classifier:libsvm_svc:gamma': 0.010312002051155041, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.02668118319520441, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.010000000000000004, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.467943082306181, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0003460616208693667, 'classifier:libsvm_svc:coef0': 0.44594844039031445, 'classifier:libsvm_svc:degree': 3},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 9389.4725511663, 'classifier:libsvm_svc:gamma': 0.0004241916739337166, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.04595035337099704, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -360,7 +430,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'none', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 238.74074979026926, 'classifier:libsvm_svc:gamma': 0.008318228094785402, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.033041992948088, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.467943082306181, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00021446470601705196, 'classifier:libsvm_svc:coef0': 0.5113342646747954, 'classifier:libsvm_svc:degree': 3},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 4448.858964396972, 'classifier:libsvm_svc:gamma': 0.0004304950432405481, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.04600478693008974, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -368,7 +438,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 261.5575012454181, 'classifier:libsvm_svc:gamma': 0.008879648982592058, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.05161911549887368, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7158114201607952, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.25, 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.513678013667205, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00021079237186779576, 'classifier:libsvm_svc:coef0': 0.5490865252846295, 'classifier:libsvm_svc:degree': 4},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 16608.037433361635, 'classifier:libsvm_svc:gamma': 0.0006444046460162696, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.034533932569194285, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -376,7 +446,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'none', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 235.8923931058993, 'classifier:libsvm_svc:gamma': 0.018997399034082388, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0007012962659233003, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.010160894056711051, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.863759655230442, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0002126471280988806, 'classifier:libsvm_svc:coef0': 0.6747151732048422, 'classifier:libsvm_svc:degree': 4},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 14859.973930048649, 'classifier:libsvm_svc:gamma': 0.00024903539301962703, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.021475389177312555, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.0067282458229984375, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -384,7 +454,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'pca', 'classifier:libsvm_svc:C': 0.9240981027125502, 'classifier:libsvm_svc:gamma': 0.008851468557757787, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 5.724426034494445e-05, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7605336648224813, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.22999843007770715, 'feature_preprocessor:pca:keep_variance': 0.9966793338180532, 'feature_preprocessor:pca:whiten': 'False', 'classifier:libsvm_svc:coef0': 0.4557419739795856, 'classifier:libsvm_svc:degree': 3},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 23879.925623096413, 'classifier:libsvm_svc:gamma': 0.0002952545856170517, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.016108827693415977, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.0025855329709703464, 'feature_preprocessor:polynomial:degree': 3, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -392,7 +462,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'fast_ica', 'classifier:libsvm_svc:C': 3.11310822225217, 'classifier:libsvm_svc:gamma': 0.009559157980168218, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0003001750845667774, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.007439796308977194, 'feature_preprocessor:fast_ica:algorithm': 'deflation', 'feature_preprocessor:fast_ica:fun': 'exp', 'feature_preprocessor:fast_ica:whiten': 'False', 'classifier:libsvm_svc:coef0': 0.4572988730246441, 'classifier:libsvm_svc:degree': 4},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 21245.927364742343, 'classifier:libsvm_svc:gamma': 0.00027682405255385775, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.07761586031454697, 'feature_preprocessor:polynomial:degree': 2, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'False'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -400,7 +470,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 0.5877420521867744, 'classifier:libsvm_svc:gamma': 0.033353015347004675, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.004038704676761408, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.80889764115218, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.0001714487468650004, 'classifier:libsvm_svc:coef0': 0.5007248102255357, 'classifier:libsvm_svc:degree': 3},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'no_preprocessing', 'classifier:libsvm_svc:C': 12923.91167672086, 'classifier:libsvm_svc:gamma': 0.028106748647672205, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 2.0706197108771777e-05, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.03972767277005085},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -408,7 +478,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'normalize', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 0.5579746801991264, 'classifier:libsvm_svc:gamma': 0.03218129260910724, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0020278510484341167, 'feature_preprocessor:liblinear_svc_preprocessor:C': 2.911135774640936, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00019994775627219102, 'classifier:libsvm_svc:coef0': 0.5022118263613939, 'classifier:libsvm_svc:degree': 3},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 1.2083257404916337, 'classifier:libsvm_svc:gamma': 0.012117661046884753, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 3.571251359430454e-05, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.04642123504473486, 'feature_preprocessor:polynomial:degree': 2, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True', 'classifier:libsvm_svc:coef0': 0.05894781195155474, 'classifier:libsvm_svc:degree': 3},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -416,7 +486,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 149.9631087537682, 'classifier:libsvm_svc:gamma': 0.023739005868634755, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0006982895001467353, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_max': 0.7535443466101083, 'data_preprocessing:numerical_transformer:rescaling:robust_scaler:q_min': 0.23043545726438344, 'feature_preprocessor:liblinear_svc_preprocessor:C': 0.6041266153712135, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.002106342813088683, 'classifier:libsvm_svc:coef0': 0.47671815360774783, 'classifier:libsvm_svc:degree': 5},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 18783.914834351766, 'classifier:libsvm_svc:gamma': 0.00010213028367780999, 'classifier:libsvm_svc:kernel': 'sigmoid', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.029731234769394273, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.010562865692611513, 'feature_preprocessor:polynomial:degree': 2, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'True', 'classifier:libsvm_svc:coef0': 0.2675160702542645},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -424,23 +494,7 @@ " 'multiclass': True,\n", " 'target_type': 'classification',\n", " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'quantile_transformer', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:libsvm_svc:C': 23.617163356857034, 'classifier:libsvm_svc:gamma': 0.03426568422270486, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0003809897288698571, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.004949301030421484, 'data_preprocessing:numerical_transformer:rescaling:quantile_transformer:n_quantiles': 847, 'data_preprocessing:numerical_transformer:rescaling:quantile_transformer:output_distribution': 'normal', 'feature_preprocessor:liblinear_svc_preprocessor:C': 3.3404000226016595, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00013812954117187317, 'classifier:libsvm_svc:coef0': 0.4249652342218557, 'classifier:libsvm_svc:degree': 2},\n", - "dataset_properties={\n", - " 'task': 2,\n", - " 'sparse': False,\n", - " 'multilabel': False,\n", - " 'multiclass': True,\n", - " 'target_type': 'classification',\n", - " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'fast_ica', 'classifier:libsvm_svc:C': 1.273540574343725, 'classifier:libsvm_svc:gamma': 0.030256779283125052, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'True', 'classifier:libsvm_svc:tol': 0.0003481986673256155, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.0035772900661613393, 'feature_preprocessor:fast_ica:algorithm': 'parallel', 'feature_preprocessor:fast_ica:fun': 'logcosh', 'feature_preprocessor:fast_ica:whiten': 'False', 'classifier:libsvm_svc:coef0': 0.8237328439746787, 'classifier:libsvm_svc:degree': 3},\n", - "dataset_properties={\n", - " 'task': 2,\n", - " 'sparse': False,\n", - " 'multilabel': False,\n", - " 'multiclass': True,\n", - " 'target_type': 'classification',\n", - " 'signed': False})),\n", - "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'standardize', 'feature_preprocessor:__choice__': 'no_preprocessing', 'classifier:libsvm_svc:C': 4.892722614575765, 'classifier:libsvm_svc:gamma': 0.011700399839412606, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.0005677621968873052, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.001670501939987889, 'classifier:libsvm_svc:coef0': 0.4572988730246441, 'classifier:libsvm_svc:degree': 3},\n", + "(0.020000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessing:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessing:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessing:numerical_transformer:imputation:strategy': 'median', 'data_preprocessing:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:libsvm_svc:C': 24259.828748453074, 'classifier:libsvm_svc:gamma': 9.729750938746538e-05, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.03488095989097569, 'data_preprocessing:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.007739246022453422, 'feature_preprocessor:polynomial:degree': 2, 'feature_preprocessor:polynomial:include_bias': 'False', 'feature_preprocessor:polynomial:interaction_only': 'True'},\n", "dataset_properties={\n", " 'task': 2,\n", " 'sparse': False,\n", @@ -452,14 +506,14 @@ "-\n", "Model Statistics\n", " auto-sklearn results:\n", - " Dataset name: d74860caaa557f473ce23908ff7ba369\n", + " Dataset name: 6c563ad8-cde6-11eb-8438-0242ac1c0002\n", " Metric: accuracy\n", " Best validation score: 0.991011\n", - " Number of target algorithm runs: 588\n", - " Number of successful target algorithm runs: 577\n", - " Number of crashed target algorithm runs: 7\n", - " Number of target algorithms that exceeded the time limit: 1\n", - " Number of target algorithms that exceeded the memory limit: 3\n", + " Number of target algorithm runs: 171\n", + " Number of successful target algorithm runs: 166\n", + " Number of crashed target algorithm runs: 1\n", + " Number of target algorithms that exceeded the time limit: 2\n", + " Number of target algorithms that exceeded the memory limit: 2\n", "\n" ] } @@ -473,7 +527,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "Ii04t6ZvO71n" }, "source": [ @@ -482,140 +535,128 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 1, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", - "id": "4qw6loMI2xej", - "outputId": "5b187e85-a54d-4856-9251-1941d10ba2b6" + "id": "Qabgrg7QTmYx", + "outputId": "5d6ed1fc-6198-4c02-a70f-51dfcdcef699" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Requirement already satisfied: autokeras==1.0.2 in /usr/local/lib/python3.6/dist-packages (1.0.2)\n", - "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from autokeras==1.0.2) (0.22.2.post1)\n", - "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from autokeras==1.0.2) (1.0.3)\n", - "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from autokeras==1.0.2) (20.4)\n", - "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from autokeras==1.0.2) (1.18.4)\n", - "Requirement already satisfied: keras-tuner>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from autokeras==1.0.2) (1.0.1)\n", - "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->autokeras==1.0.2) (1.4.1)\n", - "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->autokeras==1.0.2) (0.15.1)\n", - "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->autokeras==1.0.2) (2018.9)\n", - "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->autokeras==1.0.2) (2.8.1)\n", - "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from packaging->autokeras==1.0.2) (1.12.0)\n", - "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->autokeras==1.0.2) (2.4.7)\n", - "Requirement already satisfied: colorama in /usr/local/lib/python3.6/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (0.4.3)\n", - "Requirement already satisfied: tabulate in /usr/local/lib/python3.6/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (0.8.7)\n", - "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (0.16.0)\n", - "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (4.41.1)\n", - "Requirement already satisfied: terminaltables in /usr/local/lib/python3.6/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (3.1.0)\n", - "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (2.23.0)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner>=1.0.1->autokeras==1.0.2) (1.24.3)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner>=1.0.1->autokeras==1.0.2) (3.0.4)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner>=1.0.1->autokeras==1.0.2) (2020.4.5.1)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner>=1.0.1->autokeras==1.0.2) (2.9)\n", - "Collecting tensorflow==2.1.0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/85/d4/c0cd1057b331bc38b65478302114194bd8e1b9c2bbc06e300935c0e93d90/tensorflow-2.1.0-cp36-cp36m-manylinux2010_x86_64.whl (421.8MB)\n", - " |████████████████████████████████| 421.8MB 26kB/s \n", - "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.1.0)\n", - "Collecting gast==0.2.2\n", - " Downloading https://files.pythonhosted.org/packages/4e/35/11749bf99b2d4e3cceb4d55ca22590b0d7c2c62b9de38ac4a4a7f4687421/gast-0.2.2.tar.gz\n", - "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (3.2.1)\n", - "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.0.8)\n", - "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (0.8.1)\n", - "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (3.10.0)\n", - "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.4.1)\n", - "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (0.34.2)\n", - "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (0.9.0)\n", - "Requirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.18.4)\n", - "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.12.0)\n", - "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.29.0)\n", - "Collecting tensorflow-estimator<2.2.0,>=2.1.0rc0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/18/90/b77c328a1304437ab1310b463e533fa7689f4bfc41549593056d812fab8e/tensorflow_estimator-2.1.0-py2.py3-none-any.whl (448kB)\n", - " |████████████████████████████████| 450kB 36.3MB/s \n", - "\u001b[?25hRequirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.1.2)\n", - "Collecting tensorboard<2.2.0,>=2.1.0\n", - "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d9/41/bbf49b61370e4f4d245d4c6051dfb6db80cec672605c91b1652ac8cc3d38/tensorboard-2.1.1-py3-none-any.whl (3.8MB)\n", - " |████████████████████████████████| 3.9MB 23.9MB/s \n", - "\u001b[?25hRequirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (0.2.0)\n", - "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.12.1)\n", - "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow==2.1.0) (2.10.0)\n", - "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.8.0->tensorflow==2.1.0) (46.4.0)\n", - "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.1)\n", - "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.2.2)\n", - "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.0.1)\n", - "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.23.0)\n", - "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.7.2)\n", - "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.3.0)\n", - "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.6.0)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.0.4)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2020.4.5.1)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.9)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.24.3)\n", - "Requirement already satisfied: cachetools<3.2,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.1)\n", - "Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.0)\n", - "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.2.8)\n", - "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.0)\n", - "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.0)\n", - "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.8)\n", - "Building wheels for collected packages: gast\n", - " Building wheel for gast (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for gast: filename=gast-0.2.2-cp36-none-any.whl size=7540 sha256=064f68ae656f6e8f23f18ec601b2972d33654a60183d89d2aaab680b7cd53608\n", - " Stored in directory: /root/.cache/pip/wheels/5c/2e/7e/a1d4d4fcebe6c381f378ce7743a3ced3699feb89bcfbdadadd\n", - "Successfully built gast\n", - "ERROR: tensorflow-probability 0.10.0 has requirement gast>=0.3.2, but you'll have gast 0.2.2 which is incompatible.\n", - "Installing collected packages: gast, tensorflow-estimator, tensorboard, tensorflow\n", - " Found existing installation: gast 0.3.3\n", - " Uninstalling gast-0.3.3:\n", - " Successfully uninstalled gast-0.3.3\n", - " Found existing installation: tensorflow-estimator 2.2.0\n", - " Uninstalling tensorflow-estimator-2.2.0:\n", - " Successfully uninstalled tensorflow-estimator-2.2.0\n", - " Found existing installation: tensorboard 2.2.1\n", - " Uninstalling tensorboard-2.2.1:\n", - " Successfully uninstalled tensorboard-2.2.1\n", - " Found existing installation: tensorflow 2.2.0\n", - " Uninstalling tensorflow-2.2.0:\n", - " Successfully uninstalled tensorflow-2.2.0\n", - "Successfully installed gast-0.2.2 tensorboard-2.1.1 tensorflow-2.1.0 tensorflow-estimator-2.1.0\n" + "Requirement already satisfied: autokeras==1.0.2 in /usr/local/lib/python3.7/dist-packages (1.0.2)\n", + "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from autokeras==1.0.2) (20.9)\n", + "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from autokeras==1.0.2) (1.1.5)\n", + "Processing /root/.cache/pip/wheels/bb/a1/8a/7c3de0efb3707a1701b36ebbfdbc4e67aedf6d4943a1f463d6/keras_tuner-1.0.2-cp37-none-any.whl\n", + "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from autokeras==1.0.2) (0.24.0)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from autokeras==1.0.2) (1.19.5)\n", + "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->autokeras==1.0.2) (2.4.7)\n", + "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->autokeras==1.0.2) (2018.9)\n", + "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->autokeras==1.0.2) (2.8.1)\n", + "Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (0.8.9)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (2.23.0)\n", + "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (1.4.1)\n", + "Requirement already satisfied: terminaltables in /usr/local/lib/python3.7/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (3.1.0)\n", + "Requirement already satisfied: colorama in /usr/local/lib/python3.7/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (0.4.4)\n", + "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (0.16.0)\n", + "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from keras-tuner>=1.0.1->autokeras==1.0.2) (4.41.1)\n", + "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->autokeras==1.0.2) (2.1.0)\n", + "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->autokeras==1.0.2) (1.0.1)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->autokeras==1.0.2) (1.15.0)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->keras-tuner>=1.0.1->autokeras==1.0.2) (3.0.4)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->keras-tuner>=1.0.1->autokeras==1.0.2) (2021.5.30)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->keras-tuner>=1.0.1->autokeras==1.0.2) (2.10)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->keras-tuner>=1.0.1->autokeras==1.0.2) (1.24.3)\n", + "Installing collected packages: keras-tuner\n", + " Found existing installation: keras-tuner 1.0.0\n", + " Uninstalling keras-tuner-1.0.0:\n", + " Successfully uninstalled keras-tuner-1.0.0\n", + "Successfully installed keras-tuner-1.0.2\n", + "Requirement already satisfied: tensorflow==2.1.0 in /usr/local/lib/python3.7/dist-packages (2.1.0)\n", + "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (0.36.2)\n", + "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (3.3.0)\n", + "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (3.12.4)\n", + "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.4.1)\n", + "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.1.2)\n", + "Requirement already satisfied: tensorboard<2.2.0,>=2.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (2.1.1)\n", + "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (0.2.0)\n", + "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (0.12.0)\n", + "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (0.2.2)\n", + "Requirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.19.5)\n", + "Requirement already satisfied: tensorflow-estimator<2.2.0,>=2.1.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (2.1.0)\n", + "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.0.8)\n", + "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.34.1)\n", + "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.1.0)\n", + "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.12.1)\n", + "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (0.8.1)\n", + "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.15.0)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf>=3.8.0->tensorflow==2.1.0) (57.0.0)\n", + "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.31.0)\n", + "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.23.0)\n", + "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.0.1)\n", + "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.4)\n", + "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.3.4)\n", + "Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from keras-applications>=1.0.8->tensorflow==2.1.0) (3.1.0)\n", + "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.2.2)\n", + "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.2.8)\n", + "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3.6\" in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.7.2)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.24.3)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.10)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.0.4)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2021.5.30)\n", + "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.3.0)\n", + "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.5.0)\n", + "Requirement already satisfied: cached-property; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from h5py->keras-applications>=1.0.8->tensorflow==2.1.0) (1.5.2)\n", + "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.8)\n", + "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.1)\n", + "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.4.1)\n", + "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.7.4.3)\n", + "Collecting keras-tuner==1.0.1\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/a7/f7/4b41b6832abf4c9bef71a664dc563adb25afc5812831667c6db572b1a261/keras-tuner-1.0.1.tar.gz (54kB)\n", + "\u001b[K |████████████████████████████████| 61kB 5.7MB/s \n", + "\u001b[?25hRequirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from keras-tuner==1.0.1) (0.16.0)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from keras-tuner==1.0.1) (1.19.5)\n", + "Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from keras-tuner==1.0.1) (0.8.9)\n", + "Requirement already satisfied: terminaltables in /usr/local/lib/python3.7/dist-packages (from keras-tuner==1.0.1) (3.1.0)\n", + "Requirement already satisfied: colorama in /usr/local/lib/python3.7/dist-packages (from keras-tuner==1.0.1) (0.4.4)\n", + "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from keras-tuner==1.0.1) (4.41.1)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from keras-tuner==1.0.1) (2.23.0)\n", + "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from keras-tuner==1.0.1) (1.4.1)\n", + "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from keras-tuner==1.0.1) (0.24.0)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->keras-tuner==1.0.1) (2.10)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->keras-tuner==1.0.1) (3.0.4)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->keras-tuner==1.0.1) (2021.5.30)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->keras-tuner==1.0.1) (1.24.3)\n", + "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->keras-tuner==1.0.1) (1.0.1)\n", + "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->keras-tuner==1.0.1) (2.1.0)\n", + "Building wheels for collected packages: keras-tuner\n", + " Building wheel for keras-tuner (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for keras-tuner: filename=keras_tuner-1.0.1-cp37-none-any.whl size=73199 sha256=62bc7c1cce031f059619402a5fc4133cf9a8698cee6ebcd6082146c0fe4bacb8\n", + " Stored in directory: /root/.cache/pip/wheels/b9/cc/62/52716b70dd90f3db12519233c3a93a5360bc672da1a10ded43\n", + "Successfully built keras-tuner\n", + "Installing collected packages: keras-tuner\n", + " Found existing installation: keras-tuner 1.0.2\n", + " Uninstalling keras-tuner-1.0.2:\n", + " Successfully uninstalled keras-tuner-1.0.2\n", + "Successfully installed keras-tuner-1.0.1\n" ] - }, - { - "data": { - "application/vnd.colab-display-data+json": { - "pip_warning": { - "packages": [ - "gast", - "tensorboard", - "tensorflow" - ] - } - } - }, - "metadata": { - "tags": [] - }, - "output_type": "display_data" } ], "source": [ - "#download the necessary packages you might have to resrart your runtime after this \n", + "# Download the necessary packages. You might have to restart your runtime after this \n", "!pip install autokeras==1.0.2\n", "!pip install tensorflow==2.1.0\n", - "#!pip install git+git://github.com/keras-team/keras-tuner.git@d2d69cba21a0b482a85ce2a38893e2322e139c01" + "!pip install keras-tuner==1.0.1" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 2, "metadata": { - "colab": {}, - "colab_type": "code", "id": "uiuEn1tMQ_Sx" }, "outputs": [], @@ -623,20 +664,18 @@ "from tensorflow.keras.datasets import mnist\n", "import autokeras as ak\n", "import warnings\n", - "warnings.filterwarnings('ignore')\n" + "warnings.filterwarnings('ignore')" ] }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 3, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 68 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "uin4io1TRBpM", - "outputId": "d1762060-5126-4458-ff8b-e389ef5f3067" + "outputId": "df43895d-cb80-452a-b4bd-9b37a840b672" }, "outputs": [ { @@ -658,10 +697,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 4, "metadata": { - "colab": {}, - "colab_type": "code", "id": "nzjHV906RIMU" }, "outputs": [], @@ -677,15 +714,14 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, - "colab_type": "code", "id": "Vw1xkjmtRE_B", - "outputId": "8f8ae9b6-fc5f-4b9b-caf5-8e0702dd533a" + "outputId": "390bee20-65cf-41df-af7c-14707a9b2cb7" }, "outputs": [ { @@ -694,11 +730,11 @@ "text": [ "Train for 1500 steps, validate for 375 steps\n", "Epoch 1/3\n", - "1500/1500 [==============================] - 10s 7ms/step - loss: 0.1740 - accuracy: 0.9471 - val_loss: 0.0632 - val_accuracy: 0.9808\n", + "1500/1500 [==============================] - 9s 6ms/step - loss: 0.1764 - accuracy: 0.9460 - val_loss: 0.0666 - val_accuracy: 0.9810\n", "Epoch 2/3\n", - "1500/1500 [==============================] - 9s 6ms/step - loss: 0.0785 - accuracy: 0.9750 - val_loss: 0.0517 - val_accuracy: 0.9852\n", + "1500/1500 [==============================] - 8s 5ms/step - loss: 0.0799 - accuracy: 0.9757 - val_loss: 0.0565 - val_accuracy: 0.9827\n", "Epoch 3/3\n", - "1500/1500 [==============================] - 10s 6ms/step - loss: 0.0635 - accuracy: 0.9802 - val_loss: 0.0463 - val_accuracy: 0.9870\n" + "1500/1500 [==============================] - 8s 5ms/step - loss: 0.0636 - accuracy: 0.9801 - val_loss: 0.0516 - val_accuracy: 0.9852\n" ] }, { @@ -732,7 +768,7 @@ { "data": { "text/html": [ - " |-Trial ID: f0b82bea386c062653e6830e3cfbed5a" + " |-Trial ID: 7be96a3c496d96d460dd2af280999875" ], "text/plain": [ "" @@ -746,7 +782,7 @@ { "data": { "text/html": [ - " |-Score: 0.04631742829534535" + " |-Score: 0.051625245199888016" ], "text/plain": [ "" @@ -1043,11 +1079,11 @@ "text": [ "Train for 1500 steps, validate for 375 steps\n", "Epoch 1/3\n", - "1500/1500 [==============================] - 110s 74ms/step - loss: 0.2727 - accuracy: 0.9259 - val_loss: 0.2531 - val_accuracy: 0.9405\n", + "1500/1500 [==============================] - 68s 46ms/step - loss: 0.2376 - accuracy: 0.9351 - val_loss: 0.1227 - val_accuracy: 0.9681\n", "Epoch 2/3\n", - "1500/1500 [==============================] - 105s 70ms/step - loss: 0.1165 - accuracy: 0.9701 - val_loss: 0.1070 - val_accuracy: 0.9706\n", + "1500/1500 [==============================] - 64s 42ms/step - loss: 0.1281 - accuracy: 0.9680 - val_loss: 0.1519 - val_accuracy: 0.9637\n", "Epoch 3/3\n", - "1500/1500 [==============================] - 105s 70ms/step - loss: 0.1016 - accuracy: 0.9733 - val_loss: 0.0666 - val_accuracy: 0.9803\n" + "1500/1500 [==============================] - 63s 42ms/step - loss: 0.0949 - accuracy: 0.9742 - val_loss: 0.2995 - val_accuracy: 0.9153\n" ] }, { @@ -1081,7 +1117,7 @@ { "data": { "text/html": [ - " |-Trial ID: dbc4b7f33b7c9fe4d28bad86df312b47" + " |-Trial ID: e6126228dfef6921fd671b25eea06aff" ], "text/plain": [ "" @@ -1095,7 +1131,7 @@ { "data": { "text/html": [ - " |-Score: 0.06660779262116799" + " |-Score: 0.12271730258750418" ], "text/plain": [ "" @@ -1336,11 +1372,11 @@ "text": [ "Train for 1500 steps, validate for 375 steps\n", "Epoch 1/3\n", - "1500/1500 [==============================] - 10s 7ms/step - loss: 0.1772 - accuracy: 0.9460 - val_loss: 0.0647 - val_accuracy: 0.9817\n", + "1500/1500 [==============================] - 8s 6ms/step - loss: 0.1686 - accuracy: 0.9479 - val_loss: 0.0688 - val_accuracy: 0.9814\n", "Epoch 2/3\n", - "1500/1500 [==============================] - 9s 6ms/step - loss: 0.0770 - accuracy: 0.9753 - val_loss: 0.0542 - val_accuracy: 0.9853\n", + "1500/1500 [==============================] - 8s 5ms/step - loss: 0.0772 - accuracy: 0.9766 - val_loss: 0.0524 - val_accuracy: 0.9847\n", "Epoch 3/3\n", - "1500/1500 [==============================] - 9s 6ms/step - loss: 0.0627 - accuracy: 0.9802 - val_loss: 0.0501 - val_accuracy: 0.9857\n" + "1500/1500 [==============================] - 8s 5ms/step - loss: 0.0615 - accuracy: 0.9808 - val_loss: 0.0508 - val_accuracy: 0.9848\n" ] }, { @@ -1374,7 +1410,7 @@ { "data": { "text/html": [ - " |-Trial ID: e2507fa096f021ed66a080b76f7f392f" + " |-Trial ID: 4c463bc14ed62440099b12ff1c705258" ], "text/plain": [ "" @@ -1388,7 +1424,7 @@ { "data": { "text/html": [ - " |-Score: 0.050107761471144235" + " |-Score: 0.05078147934443162" ], "text/plain": [ "" @@ -1686,11 +1722,11 @@ "INFO:tensorflow:Oracle triggered exit\n", "Train for 1875 steps, validate for 375 steps\n", "Epoch 1/3\n", - "1875/1875 [==============================] - 12s 6ms/step - loss: 0.1570 - accuracy: 0.9519 - val_loss: 0.0474 - val_accuracy: 0.9851\n", + "1875/1875 [==============================] - 11s 6ms/step - loss: 0.1628 - accuracy: 0.9494 - val_loss: 0.0394 - val_accuracy: 0.9883\n", "Epoch 2/3\n", - "1875/1875 [==============================] - 12s 6ms/step - loss: 0.0736 - accuracy: 0.9775 - val_loss: 0.0267 - val_accuracy: 0.9926\n", + "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0744 - accuracy: 0.9780 - val_loss: 0.0266 - val_accuracy: 0.9918\n", "Epoch 3/3\n", - "1875/1875 [==============================] - 12s 6ms/step - loss: 0.0602 - accuracy: 0.9814 - val_loss: 0.0198 - val_accuracy: 0.9948\n" + "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0600 - accuracy: 0.9815 - val_loss: 0.0187 - val_accuracy: 0.9952\n" ] } ], @@ -1703,15 +1739,13 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 6, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 136 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "L2y_haC9RJvC", - "outputId": "14c6ae65-402a-4fa9-80f1-9c50e8b05c99" + "outputId": "e99afcd5-7de4-4ec6-972e-a75d25dd1f53" }, "outputs": [ { @@ -1736,15 +1770,13 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 7, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 275 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "oFSh69lSNUcX", - "outputId": "6fea2436-cacc-43f3-bc98-c450a927a774" + "outputId": "7fc93390-c71d-4aeb-e42c-677070da795b" }, "outputs": [ { @@ -1763,8 +1795,8 @@ "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay\n", "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate\n", "WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.\n", - "313/313 [==============================] - 1s 4ms/step - loss: 0.0363 - accuracy: 0.9877\n", - "[0.03626436673476883, 0.9877]\n" + "313/313 [==============================] - 1s 3ms/step - loss: 0.0386 - accuracy: 0.9865\n", + "[0.03864423465912374, 0.9865]\n" ] } ], @@ -1772,57 +1804,13 @@ "#evaluate on test data\n", "print(clf.evaluate(x_test, y_test))" ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "230VP4kTRhN-" - }, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "6MH08ZwHSBC_" - }, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "vUvyU0HaSXXT" - }, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "UyVkkN0UgdOc" - }, - "outputs": [], - "source": [] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], - "name": "AutoML Experimets.ipynb", + "name": "01_AutoML_Demo.ipynb", "provenance": [] }, "kernelspec": { @@ -1840,7 +1828,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.10" + "version": "3.6.13" } }, "nbformat": 4, diff --git a/Ch4/05_DeepNN_Example.ipynb b/Ch4/05_DeepNN_Example.ipynb index b771228..303da08 100644 --- a/Ch4/05_DeepNN_Example.ipynb +++ b/Ch4/05_DeepNN_Example.ipynb @@ -77,7 +77,7 @@ "except ModuleNotFoundError:\n", " \n", " if not os.path.exists(os.getcwd()+'\\\\Data\\\\glove.6B'):\n", - " os.makdir(os.getcwd()+'\\\\Data\\\\glove.6B')\n", + " os.mkdir(os.getcwd()+'\\\\Data\\\\glove.6B')\n", " \n", " url='http://nlp.stanford.edu/data/glove.6B.zip' \n", " path=os.getcwd()+'\\Data' \n", @@ -549,7 +549,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.0" + "version": "3.7.4" } }, "nbformat": 4, diff --git a/Ch5/01_KPE.ipynb b/Ch5/01_KPE.ipynb index 694a907..344066a 100644 --- a/Ch5/01_KPE.ipynb +++ b/Ch5/01_KPE.ipynb @@ -3,119 +3,294 @@ { "cell_type": "code", "execution_count": 1, - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "ilskamOD-PME", + "outputId": "c9bd7728-d698-446c-b784-6f3ef322ff2b" + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting textacy==0.9.1\n", - " Downloading textacy-0.9.1-py3-none-any.whl (203 kB)\n", - "\u001b[K |████████████████████████████████| 203 kB 897 kB/s eta 0:00:01\n", - "\u001b[?25hRequirement already satisfied: numpy>=1.17.0 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from textacy==0.9.1) (1.18.5)\n", - "Collecting jellyfish>=0.7.0\n", - " Downloading jellyfish-0.8.2-cp36-cp36m-manylinux2014_x86_64.whl (93 kB)\n", - "\u001b[K |████████████████████████████████| 93 kB 75 kB/s eta 0:00:01\n", - "\u001b[?25hCollecting cachetools>=2.0.1\n", - " Downloading cachetools-4.1.1-py3-none-any.whl (10 kB)\n", - "Requirement already satisfied: tqdm>=4.19.6 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from textacy==0.9.1) (4.46.1)\n", - "Requirement already satisfied: spacy>=2.0.12 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from textacy==0.9.1) (2.1.3)\n", - "Requirement already satisfied: requests>=2.10.0 in /home/etherealenvy/.local/lib/python3.6/site-packages (from textacy==0.9.1) (2.23.0)\n", - "Requirement already satisfied: scipy>=0.17.0 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from textacy==0.9.1) (1.4.1)\n", - "Requirement already satisfied: scikit-learn>=0.19.0 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from textacy==0.9.1) (0.22.2.post1)\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/3a/5e/3b8391cf6ff39350b73f8421184cf6792002b5c2c17982b7c9fbd5ff36de/textacy-0.9.1-py3-none-any.whl (203kB)\n", + "\u001b[K |████████████████████████████████| 204kB 4.1MB/s \n", + "\u001b[?25hRequirement already satisfied: pyemd>=0.5.0 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (0.5.1)\n", "Collecting cytoolz>=0.8.0\n", - " Downloading cytoolz-0.10.1.tar.gz (475 kB)\n", - "\u001b[K |████████████████████████████████| 475 kB 560 kB/s eta 0:00:01\n", - "\u001b[?25hCollecting networkx>=2.0\n", - " Using cached networkx-2.4-py3-none-any.whl (1.6 MB)\n", - "Requirement already satisfied: srsly>=0.0.5 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from textacy==0.9.1) (1.0.2)\n", - "Collecting pyemd>=0.5.0\n", - " Downloading pyemd-0.5.1.tar.gz (91 kB)\n", - "\u001b[K |████████████████████████████████| 91 kB 558 kB/s eta 0:00:011\n", - "\u001b[?25hRequirement already satisfied: joblib>=0.13.0 in /home/etherealenvy/.local/lib/python3.6/site-packages (from textacy==0.9.1) (0.14.1)\n", - "Collecting pyphen>=0.9.4\n", - " Downloading Pyphen-0.9.5-py2.py3-none-any.whl (3.0 MB)\n", - "\u001b[K |████████████████████████████████| 3.0 MB 1.3 MB/s eta 0:00:01\n", - "\u001b[?25hRequirement already satisfied: plac<1.0.0,>=0.9.6 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from spacy>=2.0.12->textacy==0.9.1) (0.9.6)\n", - "Requirement already satisfied: blis<0.3.0,>=0.2.2 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from spacy>=2.0.12->textacy==0.9.1) (0.2.4)\n", - "Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from spacy>=2.0.12->textacy==0.9.1) (2.0.1)\n", - "Requirement already satisfied: jsonschema<3.0.0,>=2.6.0 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from spacy>=2.0.12->textacy==0.9.1) (2.6.0)\n", - "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from spacy>=2.0.12->textacy==0.9.1) (2.0.3)\n", - "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from spacy>=2.0.12->textacy==0.9.1) (1.0.2)\n", - "Requirement already satisfied: thinc<7.1.0,>=7.0.2 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from spacy>=2.0.12->textacy==0.9.1) (7.0.8)\n", - "Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from spacy>=2.0.12->textacy==0.9.1) (0.6.0)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /home/etherealenvy/.local/lib/python3.6/site-packages (from requests>=2.10.0->textacy==0.9.1) (3.0.4)\n", - "Requirement already satisfied: idna<3,>=2.5 in /home/etherealenvy/.local/lib/python3.6/site-packages (from requests>=2.10.0->textacy==0.9.1) (2.9)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /home/etherealenvy/.local/lib/python3.6/site-packages (from requests>=2.10.0->textacy==0.9.1) (2020.4.5.1)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/etherealenvy/.local/lib/python3.6/site-packages (from requests>=2.10.0->textacy==0.9.1) (1.25.9)\n", - "Collecting toolz>=0.8.0\n", - " Downloading toolz-0.10.0.tar.gz (49 kB)\n", - "\u001b[K |████████████████████████████████| 49 kB 200 kB/s eta 0:00:011\n", - "\u001b[?25hRequirement already satisfied: decorator>=4.3.0 in /home/etherealenvy/miniconda3/envs/practicalnlp/lib/python3.6/site-packages (from networkx>=2.0->textacy==0.9.1) (4.4.2)\n", - "Building wheels for collected packages: cytoolz, pyemd, toolz\n", - " Building wheel for cytoolz (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Created wheel for cytoolz: filename=cytoolz-0.10.1-cp36-cp36m-linux_x86_64.whl size=1237375 sha256=b3b553b3e671dd5ee55cfd0f49b2034e46c0c83439c28c562c28c2d1c6a66d19\n", - " Stored in directory: /home/etherealenvy/.cache/pip/wheels/ae/29/ee/95cfb685f2f524674eec78a249f3439b33bbef131529eccc35\n", - " Building wheel for pyemd (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Created wheel for pyemd: filename=pyemd-0.5.1-cp36-cp36m-linux_x86_64.whl size=375283 sha256=580f4ca79efa7be7d2cd5c5e5b41bf47e8f0072fc4af1f4fd4e7edf23020e103\n", - " Stored in directory: /home/etherealenvy/.cache/pip/wheels/f9/f0/23/aefbdde40e915c67830ebecb55be2344a8b6e95fe3ce3ccf96\n", - " Building wheel for toolz (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Created wheel for toolz: filename=toolz-0.10.0-py3-none-any.whl size=55575 sha256=cda4be45740b8127f3f45a39f805bcff3be823017ffb796df4cb694e580c35e3\n", - " Stored in directory: /home/etherealenvy/.cache/pip/wheels/fc/da/2e/27e381e9cfc922d078a0a750c7ec72e76df66100e81722516d\n", - "Successfully built cytoolz pyemd toolz\n", - "Installing collected packages: jellyfish, cachetools, toolz, cytoolz, networkx, pyemd, pyphen, textacy\n", - "Successfully installed cachetools-4.1.1 cytoolz-0.10.1 jellyfish-0.8.2 networkx-2.4 pyemd-0.5.1 pyphen-0.9.5 textacy-0.9.1 toolz-0.10.0\n" + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/58/67/1c60da8ba831bfefedb64c78b9f6820bdf58972797c95644ee3191daf27a/cytoolz-0.11.0.tar.gz (477kB)\n", + "\u001b[K |████████████████████████████████| 481kB 7.0MB/s \n", + "\u001b[?25hRequirement already satisfied: numpy>=1.17.0 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (1.19.5)\n", + "Requirement already satisfied: spacy>=2.0.12 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (2.2.4)\n", + "Requirement already satisfied: scikit-learn>=0.19.0 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (0.22.2.post1)\n", + "Requirement already satisfied: requests>=2.10.0 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (2.23.0)\n", + "Requirement already satisfied: srsly>=0.0.5 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (1.0.5)\n", + "Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (2.5.1)\n", + "Requirement already satisfied: joblib>=0.13.0 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (1.0.1)\n", + "Requirement already satisfied: cachetools>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (4.2.2)\n", + "Collecting jellyfish>=0.7.0\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/30/a6/4d039bc827a102f62ce7a7910713e38fdfd7c7a40aa39c72fb14938a1473/jellyfish-0.8.2-cp37-cp37m-manylinux2014_x86_64.whl (90kB)\n", + "\u001b[K |████████████████████████████████| 92kB 5.8MB/s \n", + "\u001b[?25hCollecting pyphen>=0.9.4\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/7c/5a/5bc036e01389bc6a6667a932bac3e388de6e7fa5777a6ff50e652f60ec79/Pyphen-0.10.0-py3-none-any.whl (1.9MB)\n", + "\u001b[K |████████████████████████████████| 1.9MB 7.6MB/s \n", + "\u001b[?25hRequirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (1.4.1)\n", + "Requirement already satisfied: tqdm>=4.19.6 in /usr/local/lib/python3.7/dist-packages (from textacy==0.9.1) (4.41.1)\n", + "Requirement already satisfied: toolz>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from cytoolz>=0.8.0->textacy==0.9.1) (0.11.1)\n", + "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.0.12->textacy==0.9.1) (2.0.5)\n", + "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.0.12->textacy==0.9.1) (1.1.3)\n", + "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.0.12->textacy==0.9.1) (3.0.5)\n", + "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.0.12->textacy==0.9.1) (7.4.0)\n", + "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.0.12->textacy==0.9.1) (1.0.5)\n", + "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.0.12->textacy==0.9.1) (1.0.0)\n", + "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.0.12->textacy==0.9.1) (0.8.2)\n", + "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.0.12->textacy==0.9.1) (0.4.1)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy>=2.0.12->textacy==0.9.1) (57.0.0)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.10.0->textacy==0.9.1) (2.10)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.10.0->textacy==0.9.1) (1.24.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.10.0->textacy==0.9.1) (2020.12.5)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.10.0->textacy==0.9.1) (3.0.4)\n", + "Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.0->textacy==0.9.1) (4.4.2)\n", + "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy>=2.0.12->textacy==0.9.1) (4.0.1)\n", + "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.0.12->textacy==0.9.1) (3.7.4.3)\n", + "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.0.12->textacy==0.9.1) (3.4.1)\n", + "Building wheels for collected packages: cytoolz\n", + " Building wheel for cytoolz (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for cytoolz: filename=cytoolz-0.11.0-cp37-cp37m-linux_x86_64.whl size=1224596 sha256=51f32a9445c5176fef8fba893a1b566ecbe8cbcaa184e83ccff078b6ae3794a5\n", + " Stored in directory: /root/.cache/pip/wheels/a1/32/3c/9c9926b510647cacdde744b2c7acdf1ccd5896fbb7f8d5df0c\n", + "Successfully built cytoolz\n", + "Installing collected packages: cytoolz, jellyfish, pyphen, textacy\n", + "Successfully installed cytoolz-0.11.0 jellyfish-0.8.2 pyphen-0.10.0 textacy-0.9.1\n" ] } ], "source": [ "#We need texacy, which inturn loads spacy library\n", - "!pip install textacy==0.9.1\n", - "!python -m spacy download en_core_web_sm\n", - "\n", + "!pip install textacy==0.9.1" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "4OG9AAzE-PMI", + "outputId": "2f8cebb2-de67-46ce-eddd-aad7018aaac5" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: en_core_web_sm==2.2.5 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.5/en_core_web_sm-2.2.5.tar.gz#egg=en_core_web_sm==2.2.5 in /usr/local/lib/python3.7/dist-packages (2.2.5)\n", + "Requirement already satisfied: spacy>=2.2.2 in /usr/local/lib/python3.7/dist-packages (from en_core_web_sm==2.2.5) (2.2.4)\n", + "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (2.0.5)\n", + "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (7.4.0)\n", + "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.1.3)\n", + "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.19.5)\n", + "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (2.23.0)\n", + "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (0.8.2)\n", + "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.0.0)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (57.0.0)\n", + "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (4.41.1)\n", + "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.0.5)\n", + "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (3.0.5)\n", + "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.0.5)\n", + "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (0.4.1)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (2020.12.5)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (3.0.4)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (1.24.3)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (2.10)\n", + "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_sm==2.2.5) (4.0.1)\n", + "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_sm==2.2.5) (3.7.4.3)\n", + "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_sm==2.2.5) (3.4.1)\n", + "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n", + "You can now load the model via spacy.load('en_core_web_sm')\n" + ] + } + ], + "source": [ + "!python -m spacy download en_core_web_sm" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "OYYQEs4F-PMJ", + "outputId": "3e516c1d-c20e-4a43-b61c-7be096d31018" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: spacy==2.2.4 in /usr/local/lib/python3.7/dist-packages (2.2.4)\n", + "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (2.23.0)\n", + "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (1.0.5)\n", + "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (0.4.1)\n", + "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (1.0.0)\n", + "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (1.1.3)\n", + "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (3.0.5)\n", + "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (1.0.5)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (57.0.0)\n", + "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (4.41.1)\n", + "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (1.19.5)\n", + "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (2.0.5)\n", + "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (0.8.2)\n", + "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy==2.2.4) (7.4.0)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==2.2.4) (2.10)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==2.2.4) (1.24.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==2.2.4) (2020.12.5)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==2.2.4) (3.0.4)\n", + "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy==2.2.4) (4.0.1)\n", + "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy==2.2.4) (3.4.1)\n", + "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy==2.2.4) (3.7.4.3)\n" + ] + } + ], + "source": [ + "#Use spacy version 2.2.4 for this notebook\n", + "!pip install spacy==2.2.4" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "6fjsX2at-PMJ" + }, + "outputs": [], + "source": [ "import spacy\n", "import textacy.ke\n", - "from textacy import *\n", - "\n" + "from textacy import *" ] }, { "cell_type": "code", - "execution_count": 6, - "metadata": {}, + "execution_count": 5, + "metadata": { + "id": "hnIbu8J_-PMJ" + }, "outputs": [], "source": [ "#Load a spacy model, which will be used for all further processing.\n", - "en = textacy.load_spacy_lang(\"en_core_web_sm\")\n", - "\n", + "en = textacy.load_spacy_lang(\"en_core_web_sm\")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 73, + "resources": { + "http://localhost:8080/nbextensions/google.colab/files.js": { + "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK", + "headers": [ + [ + "content-type", + "application/javascript" + ] + ], + "ok": true, + "status": 200, + "status_text": "" + } + } + }, + "id": "ay4RCUtA-PMK", + "outputId": "164a6abc-b38f-4a9a-beac-7d3f8fa9f78f" + }, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " Upload widget is only available when the cell has been executed in the\n", + " current browser session. Please rerun this cell to enable.\n", + " \n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saving nlphistory.txt to nlphistory.txt\n" + ] + } + ], + "source": [ "#Let us use a sample text file, nlphistory.txt, which is the text from the history section of Wikipedia's\n", "#page on Natural Language Processing \n", "#https://en.wikipedia.org/wiki/Natural_language_processing\n", - "path = 'PATH TO REPO'\n", - "\n", - "mytext = open(path+'./Data/nlphistory.txt').read()\n", + "try :\n", + " from google.colab import files\n", + " uploaded=files.upload()\n", + " mytext = open('nlphistory.txt').read()\n", "\n", + "except ModuleNotFoundError :\n", + " mytext = open('Data/nlphistory.txt').read()" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "qbhMSoQd-PMK" + }, + "outputs": [], + "source": [ "#convert the text into a spacy document.\n", "doc = textacy.make_spacy_doc(mytext, lang=en)" ] }, { "cell_type": "code", - "execution_count": 7, - "metadata": {}, + "execution_count": 8, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "K83I_YH5-PML", + "outputId": "4f154541-7ac4-41a3-935e-c01756e36355" + }, "outputs": [ { "data": { "text/plain": [ - "[('successful natural language processing system', 0.025244346165829074),\n", - " ('statistical machine translation system', 0.025133044868323427),\n", - " ('natural language system', 0.02100895427515401),\n", - " ('statistical natural language processing', 0.018557133294770125),\n", - " ('natural language task', 0.01578393031285106)]" + "[('successful natural language processing system', 0.02475549496438359),\n", + " ('statistical machine translation system', 0.024648673368376665),\n", + " ('natural language system', 0.020518708001159278),\n", + " ('statistical natural language processing', 0.01858983530270439),\n", + " ('natural language task', 0.01579726776487791)]" ] }, - "execution_count": 7, - "metadata": {}, + "execution_count": 8, + "metadata": { + "tags": [] + }, "output_type": "execute_result" } ], @@ -125,8 +300,14 @@ }, { "cell_type": "code", - "execution_count": 8, - "metadata": {}, + "execution_count": 9, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "5I_-bohZ-PML", + "outputId": "d6a28b63-f35f-4487-b879-7fdeb77d2e0c" + }, "outputs": [ { "name": "stdout", @@ -146,21 +327,20 @@ }, { "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, + "execution_count": 10, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "PySOp4Os-PMM", + "outputId": "fc19f928-32f9-4b0b-c6a7-41042f33973f" + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "[{'natural language processing system'}, {'statistical machine translation'}, {'statistical model'}, {'late 1980'}, {'research'}, {'example'}, {'early'}, {'ELIZA'}, {'world'}, {'real'}]\n" + "[{'natural language processing system'}, {'statistical machine translation'}, {'statistical model'}, {'late 1980'}, {'research'}, {'example'}, {'early'}, {'world'}, {'ELIZA'}, {'real'}]\n" ] } ], @@ -173,14 +353,20 @@ }, { "cell_type": "code", - "execution_count": 8, - "metadata": {}, + "execution_count": 11, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "81sJfTis-PMM", + "outputId": "05f268ba-9b64-4ebb-f53f-5fac07710211" + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "[history, natural language processing, 1950s, work, earlier periods, Alan Turing, article, what, criterion, intelligence, Georgetown experiment, fully automatic translation, more than sixty Russian sentences, English, authors, three or five years, machine translation, solved problem.[2, real progress, ALPAC report, ten-year-long research, expectations, machine translation, Little further research, machine translation, late 1980s, first statistical machine translation systems, notably successful natural language processing systems, 1960s, SHRDLU, natural language system, restricted \"blocks worlds, restricted vocabularies, ELIZA, simulation, Rogerian psychotherapist, Joseph Weizenbaum, almost no information, human thought, emotion, ELIZA, startlingly human-like interaction, \"patient, very small knowledge base, ELIZA, generic response, example, \"My head, you, head, 1970s, many programmers, \"conceptual ontologies, real-world information, computer-understandable data, Examples, MARGIE, (Schank, SAM, (Cullingford, PAM, (Wilensky, TaleSpin, (Meehan, QUALM, (Lehnert, Politics, (Carbonell, Plot Units, Lehnert, time, many chatterbots, PARRY, Racter, Jabberwacky, 1980s, most natural language processing systems, complex sets, hand-written rules, late 1980s, revolution, natural language processing, introduction, algorithms, language processing, the steady increase, computational power, Moore's law, gradual lessening, dominance, Chomskyan theories, linguistics, e.g. transformational grammar, theoretical underpinnings, sort, corpus linguistics, machine-learning approach, language, earliest-used machine learning algorithms, decision trees, produced systems, existing hand-written rules, speech, use, hidden Markov models, natural language processing, research, statistical models, soft, probabilistic decisions, real-valued weights, features, input data, cache language models, many speech recognition systems, examples, such statistical models, Such models, unfamiliar input, errors, real-world data, more reliable results, larger system, multiple subtasks, notable early successes, field, machine translation, IBM Research, more complicated statistical models, systems, advantage, existing multilingual textual corpora, Parliament, Canada, European Union, result, laws, translation, governmental proceedings, official languages, corresponding systems, government, However, most other systems, corpora, tasks, systems, major limitation, success, systems, result, great deal, research, methods, limited amounts, data, Recent research, unsupervised and semi-supervised learning algorithms, Such algorithms, data, desired answers, combination, annotated and non-annotated data, task, supervised learning, less accurate results, given amount, input data, enormous amount, non-annotated data, other things, entire content, World Wide Web, inferior results, algorithm, low enough time complexity, representation learning, deep neural network-style machine learning methods, natural language processing, part, flurry, results, such techniques[4][5, -art, many natural language tasks, example, language, Popular techniques, use, word embeddings, semantic properties, words, increase, end, higher-level task, (e.g., question, pipeline, separate intermediate tasks, speech, dependency parsing, areas, shift, substantial changes, NLP systems, deep neural network-based approaches, new paradigm, statistical natural language processing, instance, term neural machine translation, NMT, fact, deep learning-based approaches, machine translation, sequence, need, intermediate steps, word alignment, language modeling, statistical machine translation, SMT]\n" + "[history, natural language processing, 1950s, work, earlier periods, Alan Turing, article, what, criterion, intelligence, Georgetown experiment, fully automatic translation, more than sixty Russian sentences, English, authors, three or five years, machine translation, real progress, ALPAC report, ten-year-long research, expectations, machine translation, Little further research, machine translation, late 1980s, first statistical machine translation systems, notably successful natural language processing systems, SHRDLU, natural language system, restricted \"blocks worlds, restricted vocabularies, ELIZA, simulation, Rogerian psychotherapist, Joseph Weizenbaum, almost no information, human thought, emotion, ELIZA, startlingly human-like interaction, \"patient, very small knowledge base, ELIZA, generic response, example, head, you, head, 1970s, many programmers, \"conceptual ontologies, real-world information, computer-understandable data, Examples, MARGIE, Schank, Cullingford, (Wilensky, Lehnert, (Carbonell, Lehnert, time, many chatterbots, PARRY, Racter, Jabberwacky, 1980s, most natural language processing systems, complex sets, hand-written rules, late 1980s, revolution, natural language processing, introduction, algorithms, language processing, both the steady increase, computational power, Moore's law, dominance, Chomskyan theories, linguistics, (e.g. transformational grammar, theoretical underpinnings, sort, corpus linguistics, machine-learning approach, language processing.[3, earliest-used machine learning algorithms, decision trees, produced systems, if-then rules, existing hand-written rules, speech, use, hidden Markov models, natural language processing, research, statistical models, soft, probabilistic decisions, real-valued weights, features, input data, cache language models, many speech recognition systems, examples, such statistical models, Such models, unfamiliar input, errors, real-world data, more reliable results, larger system, multiple subtasks, notable early successes, field, machine translation, IBM Research, successively more complicated statistical models, systems, advantage, existing multilingual textual corpora, Parliament, Canada, European Union, result, laws, translation, governmental proceedings, official languages, corresponding systems, government, most other systems, corpora, tasks, systems, major limitation, success, systems, result, great deal, research, methods, limited amounts, data, Recent research, unsupervised and semi-supervised learning algorithms, Such algorithms, data, desired answers, combination, annotated and non-annotated data, task, supervised learning, less accurate results, given amount, input data, enormous amount, non-annotated data, other things, entire content, World Wide Web, inferior results, algorithm, low enough time complexity, 2010s, representation learning, deep neural network-style machine learning methods, natural language processing, part, flurry, results, such techniques[4][5, -art, many natural language tasks, example, language, modeling,[6] parsing,[7][8, many others, Popular techniques, use, word embeddings, semantic properties, words, increase, end, higher-level task, (e.g., question answering, pipeline, separate intermediate tasks, speech, dependency parsing, areas, shift, substantial changes, NLP systems, deep neural network-based approaches, statistical natural language processing, instance, term neural machine translation, NMT, fact, that deep learning-based approaches, machine translation, sequence, need, intermediate steps, word alignment, language modeling, statistical machine translation, SMT]\n" ] } ], @@ -193,40 +379,40 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "id": "IryG-zS4-PMN" + }, "source": [ "Textacy also has a bunch of other information extraction functions, many of them based on regular expression patterns and heuristics to address extracting specific expressions such as acronyms and quotations. Apart from these, we can also extract matching custom regular expressions including POS tag patterns, or look for statements involving an entity, subject-verb-object tuples etc. We will discuss some of these as they come, in this chapter. \n", "\n", "Documentation: https://chartbeat-labs.github.io/textacy/build/html/index.html" ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { + "accelerator": "GPU", + "colab": { + "name": "01_KPE.ipynb", + "provenance": [] + }, "kernelspec": { - "display_name": "Python 2", + "display_name": "Python 3", "language": "python", - "name": "python2" + "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", - "version": 2 + "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.17" + "pygments_lexer": "ipython3", + "version": "3.6.13" } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 1 } diff --git a/Ch5/02_NERTraining.ipynb b/Ch5/02_NERTraining.ipynb index 3119db6..d18ccda 100644 --- a/Ch5/02_NERTraining.ipynb +++ b/Ch5/02_NERTraining.ipynb @@ -22,7 +22,7 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 1, "metadata": { "colab": {}, "colab_type": "code", @@ -37,7 +37,9 @@ "from pprint import pprint\n", "from sklearn.metrics import f1_score,classification_report\n", "from sklearn.pipeline import Pipeline\n", - "import string\n" + "import string\n", + "import warnings\n", + "warnings.filterwarnings('ignore')" ] }, { @@ -52,7 +54,7 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 2, "metadata": { "colab": {}, "colab_type": "code", @@ -79,12 +81,12 @@ " words.append(word)\n", " tags.append(tag)\n", " fh.close()\n", - " return myoutput\n" + " return myoutput" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 3, "metadata": { "colab": {}, "colab_type": "code", @@ -150,7 +152,7 @@ " wordfeats[\"nextNextTag\"] = sen_tags[i + 2][1]\n", " #That is it! You can add whatever you want!\n", " feats.append(wordfeats)\n", - " return feats\n" + " return feats" ] }, { @@ -165,7 +167,7 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 4, "metadata": { "colab": {}, "colab_type": "code", @@ -195,7 +197,7 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 5, "metadata": { "colab": {}, "colab_type": "code", @@ -217,7 +219,7 @@ " print(metrics.flat_f1_score(Y_dev, y_pred,average='weighted', labels=labels))\n", " print(metrics.flat_classification_report(Y_dev, y_pred, labels=sorted_labels, digits=3))\n", " #print(metrics.sequence_accuracy_score(Y_dev, y_pred))\n", - " get_confusion_matrix(Y_dev, y_pred,labels=sorted_labels)\n" + " get_confusion_matrix(Y_dev, y_pred,labels=sorted_labels)" ] }, { @@ -232,7 +234,7 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 6, "metadata": { "colab": {}, "colab_type": "code", @@ -258,12 +260,12 @@ " cell = \"%{0}.0f\".format(columnwidth) % cm[i, j]\n", " sum = sum + int(cell)\n", " print(cell, end=\" \")\n", - " print(sum) #Prints the total number of instances per cat at the end.\n" + " print(sum) #Prints the total number of instances per cat at the end." ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 7, "metadata": { "colab": {}, "colab_type": "code", @@ -278,7 +280,7 @@ " for yseq_true, yseq_pred in zip(y_true, y_pred):\n", " trues.extend(yseq_true)\n", " preds.extend(yseq_pred)\n", - " print_cm(confusion_matrix(trues,preds,labels),labels)\n" + " print_cm(confusion_matrix(trues,preds,labels),labels)" ] }, { @@ -293,7 +295,7 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 8, "metadata": { "colab": {}, "colab_type": "code", @@ -307,19 +309,21 @@ "text": [ "Training a Sequence classification model with CRF\n", "0.9255103670420659\n", - " precision recall f1-score support\n", + " precision recall f1-score support\n", "\n", - " O 0.973 0.981 0.977 38323\n", - " B-LOC 0.694 0.765 0.728 1668\n", - " I-LOC 0.738 0.482 0.584 257\n", - " B-MISC 0.648 0.309 0.419 702\n", - " I-MISC 0.626 0.505 0.559 216\n", - " B-ORG 0.670 0.561 0.611 1661\n", - " I-ORG 0.551 0.704 0.618 835\n", - " B-PER 0.773 0.766 0.769 1617\n", - " I-PER 0.819 0.886 0.851 1156\n", + " O 0.973 0.981 0.977 38323\n", + " B-LOC 0.694 0.765 0.728 1668\n", + " I-LOC 0.738 0.482 0.584 257\n", + " B-MISC 0.648 0.309 0.419 702\n", + " I-MISC 0.626 0.505 0.559 216\n", + " B-ORG 0.670 0.561 0.611 1661\n", + " I-ORG 0.551 0.704 0.618 835\n", + " B-PER 0.773 0.766 0.769 1617\n", + " I-PER 0.819 0.886 0.851 1156\n", "\n", - "avg / total 0.926 0.928 0.926 46435\n", + " accuracy 0.928 46435\n", + " macro avg 0.721 0.662 0.679 46435\n", + "weighted avg 0.926 0.928 0.926 46435\n", "\n", "\n", "\n", @@ -338,10 +342,18 @@ } ], "source": [ - "\n", "def main():\n", - " train_path = 'conlldata/train.txt'\n", - " test_path = 'conlldata/test.txt'\n", + " \n", + " try:\n", + " from google.colab import files\n", + " uploaded = files.upload()\n", + " # files are present in Data/conlldata\n", + " train_path = 'train.txt'\n", + " test_path = 'test.txt'\n", + " except:\n", + " train_path = 'Data/conlldata/train.txt'\n", + " test_path = 'Data/conlldata/test.txt'\n", + " \n", " conll_train = load__data_conll(train_path)\n", " conll_dev = load__data_conll(test_path)\n", " \n", @@ -352,7 +364,7 @@ " print(\"Done with sequence model\")\n", "\n", "if __name__==\"__main__\":\n", - " main()\n" + " main()" ] }, { @@ -364,17 +376,6 @@ "source": [ "This is pretty good. We already have a model which has an F-score of 92%!!!" ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "reO-wnZocnB2" - }, - "outputs": [], - "source": [] } ], "metadata": { @@ -383,21 +384,21 @@ "provenance": [] }, "kernelspec": { - "display_name": "Python 2", + "display_name": "Python 3", "language": "python", - "name": "python2" + "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", - "version": 2 + "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.17" + "pygments_lexer": "ipython3", + "version": "3.7.4" } }, "nbformat": 4, diff --git a/Ch5/04_NER_using_spaCy - CoNLL.ipynb b/Ch5/04_NER_using_spaCy - CoNLL.ipynb index e73567a..d4d128d 100644 --- a/Ch5/04_NER_using_spaCy - CoNLL.ipynb +++ b/Ch5/04_NER_using_spaCy - CoNLL.ipynb @@ -1,336 +1,606 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Training and Evaluating an NER model with spaCy on the CoNLL dataset\n", - "\n", - "In this notebook, we will take a look at using spaCy commandline to train and evaluate a NER model. We will also compare it with the pretrained NER model in spacy. \n", - "\n", - "Note: we will create multiple folders during this experiment:\n", - "spacyNER_data " - ] + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "name": "04_NER_using_spaCy_CoNLL.ipynb", + "provenance": [], + "collapsed_sections": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.3" + }, + "accelerator": "GPU" }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 1: Converting data to json structures so it can be used by Spacy" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ + "cells": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "mkdir: cannot create directory ‘spacyNER_data’: File exists\n", - "\u001b[38;5;2m✔ Generated output file (1 documents)\u001b[0m\n", - "spacyNER_data/train.json\n", - "\u001b[38;5;2m✔ Generated output file (1 documents)\u001b[0m\n", - "spacyNER_data/test.json\n", - "\u001b[38;5;2m✔ Generated output file (1 documents)\u001b[0m\n", - "spacyNER_data/valid.json\n" - ] - } - ], - "source": [ - "#Read the CONLL data from conll2003 folder, and store the formatted data into a folder spacyNER_data\n", - "!mkdir spacyNER_data\n", - "#the above two lines create folders if they don't exist. If they do, the output shows a message that it\n", - "#already exists and cannot be created again\n", - "!python3 -m spacy convert \"Data/conll2003/en/train.txt\" spacyNER_data -c ner\n", - "!python3 -m spacy convert \"Data/conll2003/en/test.txt\" spacyNER_data -c ner\n", - "!python3 -m spacy convert \"Data/conll2003/en/valid.txt\" spacyNER_data -c ner" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### For example, the data before and after running spacy's convert program looks as follows." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": { - "scrolled": true - }, - "outputs": [ + "cell_type": "markdown", + "metadata": { + "id": "q5D0DhOQwTwq" + }, + "source": [ + "# Training and Evaluating an NER model with spaCy on the CoNLL dataset\n", + "\n", + "In this notebook, we will take a look at using spaCy commandline to train and evaluate a NER model. We will also compare it with the pretrained NER model in spacy. \n", + "\n", + "Note: we will create multiple folders during this experiment:\n", + "spacyNER_data " + ] + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "BEFORE : (Data/conll2003/en/train.txt)\n", - "EU NNP B-NP B-ORG\n", - "rejects VBZ B-VP O\n", - "German JJ B-NP B-MISC\n", - "call NN I-NP O\n", - "to TO B-VP O\n", - "boycott VB I-VP O\n", - "British JJ B-NP B-MISC\n", - "lamb NN I-NP O\n", - ". . O O\n", - "\n", - "AFTER : (Data/conll2003/en/train.json)\n", - " {\n", - " \"tokens\":[\n", - " {\n", - " \"orth\":\"EU\",\n", - " \"tag\":\"NNP\",\n", - " \"ner\":\"U-ORG\"\n", - " },\n", - " {\n", - " \"orth\":\"rejects\",\n", - " \"tag\":\"VBZ\",\n", - " \"ner\":\"O\"\n", - " },\n", - " {\n", - " \"orth\":\"German\",\n", - " \"tag\":\"JJ\",\n", - " \"ner\":\"U-MISC\"\n", - " },\n", - " {\n", - " \"orth\":\"call\",\n", - " \"tag\":\"NN\",\n", - " \"ner\":\"O\"\n", - " },\n", - " {\n", - " \"orth\":\"to\",\n", - " \"tag\":\"TO\",\n", - " \"ner\":\"O\"\n", - " },\n", - " {\n", - " \"orth\":\"boycott\",\n", - " \"tag\":\"VB\",\n", - " \"ner\":\"O\"\n", - " },\n", - " {\n", - " \"orth\":\"British\",\n", - " \"tag\":\"JJ\",\n", - " \"ner\":\"U-MISC\"\n", - " },\n", - " {\n", - " \"orth\":\"lamb\",\n", - " \"tag\":\"NN\",\n", - " \"ner\":\"O\"\n", - " },\n", - " {\n", - " \"orth\":\".\",\n", - " \"tag\":\".\",\n", - " \"ner\":\"O\"\n", - " }\n", - " ]\n", - " },\n" - ] - } - ], - "source": [ - "!echo \"BEFORE : (Data/conll2003/en/train.txt)\"\n", - "!head \"Data/conll2003/en/train.txt\" -n 11 | tail -n 9\n", - "!echo \"\\nAFTER : (Data/conll2003/en/train.json)\"\n", - "!head \"spacyNER_data/train.json\" -n 64 | tail -n 49" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Training the NER model with Spacy (CLI)\n", - "\n", - "All the commandline options can be seen at: https://spacy.io/api/cli#train\n", - "We are training using the train program in spacy, for English (en), and the results are stored in a folder \n", - "called \"model\" (created while training). Our training file is in \"spacyNER_data/train.json\" and the validation file is at: \"spacyNER_data/valid.json\". \n", - "\n", - "-G stands for gpu option.\n", - "-p stands for pipeline, and it should be followed by a comma separated set of options - in this case, a tagger and an NER are being trained simultaneously" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ + "cell_type": "markdown", + "metadata": { + "id": "AMQsjIY_wTwu" + }, + "source": [ + "## Step 1: Converting data to json structures so it can be used by Spacy" + ] + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "Training pipeline: ['tagger', 'ner']\n", - "Starting with blank model 'en'\n", - "Counting training words (limit=0)\n", - "\n", - "Itn Dep Loss NER Loss UAS NER P NER R NER F Tag % Token % CPU WPS GPU WPS\n", - "--- ---------- ---------- ------- ------- ------- ------- ------- ------- ------- -------\n", - " 0 0.000 20994.512 0.000 78.404 77.230 77.813 94.075 100.000 15468 0\n", - " 1 0.000 10338.546 0.000 84.808 84.366 84.586 94.812 100.000 15833 0\n", - " 2 0.000 7414.531 0.000 86.235 85.931 86.083 95.015 100.000 15839 0\n", - " 3 0.000 5461.594 0.000 87.020 86.873 86.946 95.106 100.000 15737 0\n", - " 4 0.000 4101.375 0.000 87.669 87.344 87.506 95.182 100.000 15887 0\n", - " 5 0.000 3413.915 0.000 87.622 87.327 87.475 95.258 100.000 15919 0\n", - " 6 0.000 3008.749 0.000 88.024 87.580 87.802 95.322 100.000 18794 0\n", - " 7 0.000 2704.280 0.000 88.323 87.832 88.077 95.347 100.000 15652 0\n", - " 8 0.000 2301.952 0.000 88.195 87.883 88.038 95.405 100.000 15935 0\n", - " 9 0.000 2162.503 0.000 88.227 88.034 88.131 95.428 100.000 15866 0\n", - " 10 0.000 1954.655 0.000 88.394 88.186 88.290 95.409 100.000 15689 0\n", - " 11 0.000 1846.583 0.000 88.233 88.085 88.159 95.391 100.000 15812 0\n", - " 12 0.000 1760.181 0.000 88.682 88.354 88.518 95.452 100.000 15829 0\n", - " 13 0.000 1670.751 0.000 88.579 88.236 88.407 95.465 100.000 15689 0\n", - " 14 0.000 1534.231 0.000 88.443 88.219 88.331 95.481 100.000 15662 0\n", - " 15 0.000 1439.400 0.000 88.782 88.438 88.610 95.510 100.000 15864 0\n", - " 16 0.000 1407.665 0.000 88.915 88.556 88.735 95.477 100.000 15872 0\n", - " 17 0.000 1199.285 0.000 88.709 88.455 88.582 95.512 100.000 15826 0\n", - " 18 0.000 1302.530 0.000 88.709 88.455 88.582 95.512 100.000 15776 0\n", - " 19 0.000 1147.754 0.000 88.874 88.455 88.664 95.519 100.000 19138 0\n", - " 20 0.000 1115.887 0.000 88.987 88.388 88.686 95.519 100.000 19035 0\n", - " 21 0.000 1146.815 0.000 89.006 88.421 88.713 95.531 100.000 15839 0\n", - " 22 0.000 1143.363 0.000 89.122 88.522 88.821 95.529 100.000 15981 0\n", - " 23 0.000 1051.906 0.000 89.171 88.556 88.863 95.550 100.000 15931 0\n", - " 24 0.000 922.404 0.000 89.124 88.674 88.898 95.550 100.000 16115 0\n", - " 25 0.000 1033.210 0.000 89.013 88.758 88.885 95.527 100.000 15973 0\n", - " 26 0.000 939.757 0.000 88.962 88.708 88.835 95.539 100.000 15939 0\n", - " 27 0.000 874.334 0.000 88.808 88.539 88.674 95.521 100.000 15963 0\n", - " 28 0.000 847.320 0.000 88.870 88.691 88.780 95.541 100.000 15855 0\n", - " 29 0.000 879.595 0.000 88.763 88.674 88.719 95.564 100.000 15893 0\n", - "\u001b[38;5;2m✔ Saved model to output directory\u001b[0m\n", - "model/model-final\n", - "\u001b[2K\u001b[38;5;2m✔ Created best model\u001b[0m\n", - "model/model-best\n" - ] - } - ], - "source": [ - "!python3 -m spacy train en model spacyNER_data/train.json spacyNER_data/valid.json -G -p tagger,ner" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice how the performance improves with each iteration!\n", - "## Evaluating the model with test data set (`spacyNER_data/test.json`)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### On Trained model (`model/model-best`)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ + "cell_type": "code", + "metadata": { + "id": "0ENm_PG8wTwu" + }, + "source": [ + "import os" + ], + "execution_count": 1, + "outputs": [] + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[1m\n", - "================================== Results ==================================\u001b[0m\n", - "\n", - "Time 3.53 s\n", - "Words 46666 \n", - "Words/s 13234 \n", - "TOK 100.00\n", - "POS 94.79 \n", - "UAS 0.00 \n", - "LAS 0.00 \n", - "NER P 78.09 \n", - "NER R 78.75 \n", - "NER F 78.42 \n", - "\n", - "\u001b[38;5;2m✔ Generated 25 parses as HTML\u001b[0m\n", - "result\n" - ] - } - ], - "source": [ - "#create a folder to store the output and visualizations. \n", - "!mkdir result\n", - "!python3 -m spacy evaluate model/model-best spacyNER_data/test.json -dp result\n", - "# !python -m spacy evaluate model/model-final data/test.txt.json -dp result" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "a Visualization of the entity tagged test data can be seen in result/entities.html folder. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### On spacy's Pretrained NER model (`en`)" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ + "cell_type": "code", + "metadata": { + "colab": { + "resources": { + "http://localhost:8080/nbextensions/google.colab/files.js": { + "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK", + "ok": true, + "headers": [ + [ + "content-type", + "application/javascript" + ] + ], + "status": 200, + "status_text": "" + } + }, + "base_uri": "https://localhost:8080/", + "height": 140 + }, + "id": "bvKsy7lSwTwv", + "outputId": "d5f9c665-76ab-4d5a-fce3-2183be45c369" + }, + "source": [ + "# upload train.txt, test.txt, valid.txt from Data/conll2003/en\n", + "try:\n", + " from google.colab import files\n", + " uploaded = files.upload()\n", + "except ModuleNotFoundError:\n", + " print('Not using colab')" + ], + "execution_count": 2, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " Upload widget is only available when the cell has been executed in the\n", + " current browser session. Please rerun this cell to enable.\n", + " \n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + } + }, + { + "output_type": "stream", + "text": [ + "Saving test.txt to test.txt\n", + "Saving train.txt to train.txt\n", + "Saving valid.txt to valid.txt\n" + ], + "name": "stdout" + } + ] + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[1m\n", - "================================== Results ==================================\u001b[0m\n", - "\n", - "Time 6.52 s\n", - "Words 46666 \n", - "Words/s 7160 \n", - "TOK 100.00\n", - "POS 86.84 \n", - "UAS 0.00 \n", - "LAS 0.00 \n", - "NER P 7.97 \n", - "NER R 10.68 \n", - "NER F 9.12 \n", - "\n", - "\u001b[38;5;2m✔ Generated 25 parses as HTML\u001b[0m\n", - "pretrained_result\n" - ] + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "g-KC1D-ZwTwx", + "outputId": "b4a9227e-e439-4aa4-cd75-2c3a7664e45a" + }, + "source": [ + "#Read the CONLL data from conll2003 folder, and store the formatted data into a folder spacyNER_data\n", + "\n", + "# !mkdir spacyNER_data\n", + "os.mkdir('spacyNER_data')\n", + " \n", + "#the above lines create folder if it doesn't exist. If it does, the output shows a message that it\n", + "#already exists and cannot be created again\n", + "try:\n", + " import google.colab \n", + " !python -m spacy convert \"train.txt\" spacyNER_data -c ner\n", + " !python -m spacy convert \"test.txt\" spacyNER_data -c ner\n", + " !python -m spacy convert \"valid.txt\" spacyNER_data -c ner\n", + "except ModuleNotFoundError:\n", + " !python -m spacy convert \"Data/conll2003/en/train.txt\" spacyNER_data -c ner\n", + " !python -m spacy convert \"Data/conll2003/en/test.txt\" spacyNER_data -c ner\n", + " !python -m spacy convert \"Data/conll2003/en/valid.txt\" spacyNER_data -c ner" + ], + "execution_count": 3, + "outputs": [ + { + "output_type": "stream", + "text": [ + "\u001b[38;5;4mℹ Auto-detected token-per-line NER format\u001b[0m\n", + "\u001b[38;5;4mℹ Grouping every 1 sentences into a document.\u001b[0m\n", + "\u001b[38;5;3m⚠ To generate better training data, you may want to group sentences\n", + "into documents with `-n 10`.\u001b[0m\n", + "\u001b[38;5;2m✔ Generated output file (14987 documents): spacyNER_data/train.json\u001b[0m\n", + "\u001b[38;5;4mℹ Auto-detected token-per-line NER format\u001b[0m\n", + "\u001b[38;5;4mℹ Grouping every 1 sentences into a document.\u001b[0m\n", + "\u001b[38;5;3m⚠ To generate better training data, you may want to group sentences\n", + "into documents with `-n 10`.\u001b[0m\n", + "\u001b[38;5;2m✔ Generated output file (3684 documents): spacyNER_data/test.json\u001b[0m\n", + "\u001b[38;5;4mℹ Auto-detected token-per-line NER format\u001b[0m\n", + "\u001b[38;5;4mℹ Grouping every 1 sentences into a document.\u001b[0m\n", + "\u001b[38;5;3m⚠ To generate better training data, you may want to group sentences\n", + "into documents with `-n 10`.\u001b[0m\n", + "\u001b[38;5;2m✔ Generated output file (3466 documents): spacyNER_data/valid.json\u001b[0m\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SMI4n0JywTwx" + }, + "source": [ + "#### For example, the data before and after running spacy's convert program looks as follows." + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "X4wBa1MGwTwy", + "outputId": "7d07c574-02e7-470b-b3c7-bd29b0c4c3d6" + }, + "source": [ + "try:\n", + " import google.colab\n", + " !echo \"BEFORE : (train.txt)\"\n", + " !head \"train.txt\" -n 11 | tail -n 9\n", + "except ModuleNotFoundError:\n", + " print(\"BEFORE : (Data/conll2003/en/train.txt)\")\n", + " file = open(\"Data/conll2003/en/train.txt\")\n", + " content = file.readlines()\n", + " print(*content[1:11])" + ], + "execution_count": 4, + "outputs": [ + { + "output_type": "stream", + "text": [ + "BEFORE : (train.txt)\n", + "EU NNP B-NP B-ORG\n", + "rejects VBZ B-VP O\n", + "German JJ B-NP B-MISC\n", + "call NN I-NP O\n", + "to TO B-VP O\n", + "boycott VB I-VP O\n", + "British JJ B-NP B-MISC\n", + "lamb NN I-NP O\n", + ". . O O\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "oDYEtB09wTwy", + "outputId": "e6165b85-2b17-4323-ad37-2e54c4c2d629" + }, + "source": [ + "try:\n", + " import google.colab\n", + " !echo \"AFTER : (spacyNER_data/train.json)\"\n", + " !head \"spacyNER_data/train.json\" -n 77 | tail -n 58\n", + "except ModuleNotFoundError:\n", + " print(\"AFTER : (spacyNER_data/train.json)\")\n", + " f = open('spacyNER_data/train.json')\n", + " content = f.readlines()\n", + " print(*content[19:77])" + ], + "execution_count": 5, + "outputs": [ + { + "output_type": "stream", + "text": [ + "AFTER : (spacyNER_data/train.json)\n", + " {\n", + " \"id\":1,\n", + " \"paragraphs\":[\n", + " {\n", + " \"sentences\":[\n", + " {\n", + " \"tokens\":[\n", + " {\n", + " \"orth\":\"EU\",\n", + " \"tag\":\"NNP\",\n", + " \"ner\":\"U-ORG\"\n", + " },\n", + " {\n", + " \"orth\":\"rejects\",\n", + " \"tag\":\"VBZ\",\n", + " \"ner\":\"O\"\n", + " },\n", + " {\n", + " \"orth\":\"German\",\n", + " \"tag\":\"JJ\",\n", + " \"ner\":\"U-MISC\"\n", + " },\n", + " {\n", + " \"orth\":\"call\",\n", + " \"tag\":\"NN\",\n", + " \"ner\":\"O\"\n", + " },\n", + " {\n", + " \"orth\":\"to\",\n", + " \"tag\":\"TO\",\n", + " \"ner\":\"O\"\n", + " },\n", + " {\n", + " \"orth\":\"boycott\",\n", + " \"tag\":\"VB\",\n", + " \"ner\":\"O\"\n", + " },\n", + " {\n", + " \"orth\":\"British\",\n", + " \"tag\":\"JJ\",\n", + " \"ner\":\"U-MISC\"\n", + " },\n", + " {\n", + " \"orth\":\"lamb\",\n", + " \"tag\":\"NN\",\n", + " \"ner\":\"O\"\n", + " },\n", + " {\n", + " \"orth\":\".\",\n", + " \"tag\":\".\",\n", + " \"ner\":\"O\"\n", + " }\n", + " ]\n", + " }\n", + " ]\n", + " }\n", + " ]\n", + " },\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bqDOcC5OwTwz" + }, + "source": [ + "## Training the NER model with Spacy (CLI)\n", + "\n", + "All the commandline options can be seen at: https://spacy.io/api/cli#train\n", + "We are training using the train program in spacy, for English (en), and the results are stored in a folder \n", + "called \"model\" (created while training). Our training file is in \"spacyNER_data/train.json\" and the validation file is at: \"spacyNER_data/valid.json\". \n", + "\n", + "-G stands for gpu option.\n", + "-p stands for pipeline, and it should be followed by a comma separated set of options - in this case, a tagger and an NER are being trained simultaneously" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "QCmQLlwAwTw0", + "outputId": "3fba6a2a-528e-41c1-c1d4-48f64d07a1aa" + }, + "source": [ + "!python -m spacy train en model spacyNER_data/train.json spacyNER_data/valid.json -G -p tagger,ner" + ], + "execution_count": 6, + "outputs": [ + { + "output_type": "stream", + "text": [ + "\u001b[38;5;2m✔ Created output directory: model\u001b[0m\n", + "Training pipeline: ['tagger', 'ner']\n", + "Starting with blank model 'en'\n", + "Counting training words (limit=0)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W022] Training a new part-of-speech tagger using a model with no lemmatization rules or data. This means that the trained model may not be able to lemmatize correctly. If this is intentional or the language you're using doesn't have lemmatization data, you can ignore this warning by setting SPACY_WARNING_IGNORE=W022. If this is surprising, make sure you have the spacy-lookups-data package installed.\n", + " \"__main__\", mod_spec)\n", + "\n", + "Itn Tag Loss Tag % NER Loss NER P NER R NER F Token % CPU WPS\n", + "--- --------- -------- --------- ------ ------ ------ ------- -------\n", + " 1 31357.874 94.123 16637.228 83.622 82.750 83.184 100.000 12649\n", + " 2 16652.860 94.740 7563.159 86.609 85.880 86.243 100.000 12494\n", + " 3 13636.214 95.052 5209.640 87.165 86.520 86.841 100.000 12450\n", + " 4 11891.992 95.196 3864.242 88.239 87.630 87.934 100.000 12479\n", + " 5 10460.735 95.258 2959.224 88.616 87.900 88.256 100.000 12095\n", + " 6 9587.853 95.370 2600.690 88.645 87.765 88.203 100.000 12602\n", + " 7 9025.242 95.424 2229.027 88.305 87.681 87.992 100.000 12351\n", + " 8 8356.277 95.459 1983.831 88.396 87.816 88.105 100.000 12480\n", + " 9 7938.872 95.527 1891.408 88.505 87.984 88.244 100.000 12750\n", + " 10 7525.471 95.595 1646.012 88.458 87.967 88.212 100.000 12425\n", + " 11 7109.750 95.605 1616.812 88.238 87.748 87.993 100.000 12449\n", + " 12 6902.928 95.649 1613.600 88.209 87.630 87.919 100.000 12736\n", + " 13 6548.162 95.675 1354.859 88.088 87.614 87.850 100.000 12523\n", + " 14 6241.111 95.682 1340.341 88.148 87.614 87.880 100.000 12433\n", + " 15 6061.177 95.676 1257.841 88.272 87.782 88.026 100.000 12374\n", + " 16 5861.997 95.669 1227.492 88.481 88.034 88.257 100.000 12612\n", + " 17 5622.559 95.605 1130.777 88.573 88.051 88.311 100.000 12712\n", + " 18 5395.814 95.636 1034.316 88.413 87.967 88.190 100.000 12828\n", + " 19 5191.170 95.655 1049.293 88.166 87.765 87.965 100.000 12681\n", + " 20 5114.974 95.649 1013.947 88.361 87.900 88.130 100.000 12713\n", + " 21 4913.621 95.636 1067.794 88.305 87.933 88.119 100.000 12578\n", + " 22 4744.514 95.643 996.963 88.380 87.933 88.156 100.000 12627\n", + " 23 4592.323 95.645 896.812 88.473 87.967 88.219 100.000 12656\n", + " 24 4522.925 95.642 908.710 88.402 87.866 88.133 100.000 12687\n", + " 25 4336.485 95.618 847.692 88.347 87.782 88.063 100.000 12400\n", + " 26 4331.800 95.628 795.431 88.409 87.933 88.171 100.000 12504\n", + " 27 4182.228 95.640 842.904 88.344 87.883 88.113 100.000 12669\n", + " 28 4021.268 95.657 763.137 88.395 87.933 88.163 100.000 12654\n", + " 29 3982.042 95.665 805.362 88.413 87.967 88.190 100.000 12840\n", + " 30 3947.585 95.638 681.912 88.464 88.018 88.240 100.000 12514\n", + "\u001b[38;5;2m✔ Saved model to output directory\u001b[0m\n", + "model/model-final\n", + "\u001b[2K\u001b[38;5;2m✔ Created best model\u001b[0m\n", + "model/model-best\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NWj_XbNtwTw0" + }, + "source": [ + "Notice how the performance improves with each iteration!\n", + "## Evaluating the model with test data set (`spacyNER_data/test.json`)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2_zTrgq2wTw1" + }, + "source": [ + "### On Trained model (`model/model-best`)" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "dxuTnZoVwTw1", + "outputId": "a44c3bb8-5268-40fc-c40a-c9bcf1a496d8" + }, + "source": [ + "#create a folder to store the output and visualizations. \n", + "# !mkdir result\n", + "os.mkdir('result')\n", + "!python -m spacy evaluate model/model-best spacyNER_data/test.json -dp result\n", + "# !python -m spacy evaluate model/model-final data/test.txt.json -dp result" + ], + "execution_count": 7, + "outputs": [ + { + "output_type": "stream", + "text": [ + "\u001b[1m\n", + "================================== Results ==================================\u001b[0m\n", + "\n", + "Time 3.93 s\n", + "Words 46666 \n", + "Words/s 11873 \n", + "TOK 100.00\n", + "POS 95.28 \n", + "UAS 0.00 \n", + "LAS 0.00 \n", + "NER P 81.80 \n", + "NER R 81.96 \n", + "NER F 81.88 \n", + "Textcat 0.00 \n", + "\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "\u001b[38;5;2m✔ Generated 25 parses as HTML\u001b[0m\n", + "result\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4DHk91sbwTw1" + }, + "source": [ + "a Visualization of the entity tagged test data can be seen in result/entities.html folder. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZRGGHohpwTw2" + }, + "source": [ + "### On spacy's Pretrained NER model (`en_core_web_sm`)" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "T0gCT9f-iwjN", + "outputId": "0c3670e2-8e6c-4234-9d5c-16e06fd1e7bf" + }, + "source": [ + "!python -m spacy download en_core_web_sm" + ], + "execution_count": 8, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Requirement already satisfied: en_core_web_sm==2.2.5 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.5/en_core_web_sm-2.2.5.tar.gz#egg=en_core_web_sm==2.2.5 in /usr/local/lib/python3.7/dist-packages (2.2.5)\n", + "Requirement already satisfied: spacy>=2.2.2 in /usr/local/lib/python3.7/dist-packages (from en_core_web_sm==2.2.5) (2.2.4)\n", + "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (0.4.1)\n", + "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (3.0.5)\n", + "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (2.0.5)\n", + "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.0.5)\n", + "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (4.41.1)\n", + "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.1.3)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (57.0.0)\n", + "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (2.23.0)\n", + "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.19.5)\n", + "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.0.5)\n", + "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (7.4.0)\n", + "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.0.0)\n", + "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (0.8.2)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (2.10)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (3.0.4)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (2021.5.30)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (1.24.3)\n", + "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_sm==2.2.5) (4.5.0)\n", + "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_sm==2.2.5) (3.7.4.3)\n", + "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_sm==2.2.5) (3.4.1)\n", + "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n", + "You can now load the model via spacy.load('en_core_web_sm')\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "MMnN80NDwTw2", + "outputId": "e5f7b022-ff6e-426b-d471-1d101b2249fd" + }, + "source": [ + "# !mkdir pretrained_result\n", + "os.mkdir('pretrained_result')\n", + "!python -m spacy evaluate en_core_web_sm spacyNER_data/test.json -dp pretrained_result" + ], + "execution_count": 9, + "outputs": [ + { + "output_type": "stream", + "text": [ + "\u001b[1m\n", + "================================== Results ==================================\u001b[0m\n", + "\n", + "Time 7.19 s\n", + "Words 46666 \n", + "Words/s 6490 \n", + "TOK 100.00\n", + "POS 86.21 \n", + "UAS 0.00 \n", + "LAS 0.00 \n", + "NER P 6.51 \n", + "NER R 9.17 \n", + "NER F 7.62 \n", + "Textcat 0.00 \n", + "\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "/usr/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n", + " \"__main__\", mod_spec)\n", + "\u001b[38;5;2m✔ Generated 25 parses as HTML\u001b[0m\n", + "pretrained_result\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6GiGnqnpwTw3" + }, + "source": [ + "a Visualization of the entity tagged test data can be seen in pretrained_result/entities.html folder. " + ] } - ], - "source": [ - "!mkdir pretrained_result\n", - "!python3 -m spacy evaluate en spacyNER_data/test.json -dp pretrained_result" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "a Visualization of the entity tagged test data can be seen in pretrained_result/entities.html folder. " - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 2", - "language": "python", - "name": "python2" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 2 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.17" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} + ] +} \ No newline at end of file diff --git a/Ch5/05_BERT_CONLL_NER.ipynb b/Ch5/05_BERT_CONLL_NER.ipynb index ebd78ab..9f0745a 100644 --- a/Ch5/05_BERT_CONLL_NER.ipynb +++ b/Ch5/05_BERT_CONLL_NER.ipynb @@ -3,7 +3,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "SFcnXxiQX-bS" }, "source": [ @@ -16,71 +15,77 @@ "execution_count": 1, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 749 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "ZNKhShOaylVt", - "outputId": "f759624b-045b-46c9-feb9-35cdc603d356" + "outputId": "8c06a549-62a4-42dc-df3f-d727d625c557" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "TensorFlow 1.x selected.\n", "Collecting pytorch-pretrained-bert==0.4.0\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/95/68/84de54aea460eb5b2e90bf47a429aacc1ce97ff052ec40874ea38ae2331d/pytorch_pretrained_bert-0.4.0-py3-none-any.whl (45kB)\n", - "\u001b[K |████████████████████████████████| 51kB 4.0MB/s \n", - "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (2.23.0)\n", - "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.13.13)\n", - "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (4.41.1)\n", - "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.18.4)\n", - "Requirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.5.0+cu101)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (2.9)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (3.0.4)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (1.24.3)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (2020.4.5.1)\n", - "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert==0.4.0) (0.3.3)\n", - "Requirement already satisfied: botocore<1.17.0,>=1.16.13 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert==0.4.0) (1.16.13)\n", - "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert==0.4.0) (0.10.0)\n", - "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch>=0.4.1->pytorch-pretrained-bert==0.4.0) (0.16.0)\n", - "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.13->boto3->pytorch-pretrained-bert==0.4.0) (2.8.1)\n", - "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.13->boto3->pytorch-pretrained-bert==0.4.0) (0.15.2)\n", - "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.17.0,>=1.16.13->boto3->pytorch-pretrained-bert==0.4.0) (1.12.0)\n", - "Installing collected packages: pytorch-pretrained-bert\n", - "Successfully installed pytorch-pretrained-bert-0.4.0\n", + "\r", + "\u001b[K |███████▎ | 10kB 23.3MB/s eta 0:00:01\r", + "\u001b[K |██████████████▌ | 20kB 18.0MB/s eta 0:00:01\r", + "\u001b[K |█████████████████████▊ | 30kB 15.0MB/s eta 0:00:01\r", + "\u001b[K |█████████████████████████████ | 40kB 14.0MB/s eta 0:00:01\r", + "\u001b[K |████████████████████████████████| 51kB 3.9MB/s \n", + "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert==0.4.0) (4.41.1)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.19.5)\n", + "Collecting boto3\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/1b/55/e66b557bdbc266ab4f15249f382f5d7d165fee1caa7e12c96348c05ea53d/boto3-1.17.95-py2.py3-none-any.whl (131kB)\n", + "\u001b[K |████████████████████████████████| 133kB 14.2MB/s \n", + "\u001b[?25hRequirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.8.1+cu101)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert==0.4.0) (2.23.0)\n", + "Collecting botocore<1.21.0,>=1.20.95\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/c9/70/43ebe47ab7591eca35be1dcbb0e4ede2a3cf15915ad05876920b63296988/botocore-1.20.95-py2.py3-none-any.whl (7.6MB)\n", + "\u001b[K |████████████████████████████████| 7.6MB 22.4MB/s \n", + "\u001b[?25hCollecting s3transfer<0.5.0,>=0.4.0\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/63/d0/693477c688348654ddc21dcdce0817653a294aa43f41771084c25e7ff9c7/s3transfer-0.4.2-py2.py3-none-any.whl (79kB)\n", + "\u001b[K |████████████████████████████████| 81kB 9.3MB/s \n", + "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n", + " Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n", + "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=0.4.1->pytorch-pretrained-bert==0.4.0) (3.7.4.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (2021.5.30)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (1.24.3)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (2.10)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (3.0.4)\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.21.0,>=1.20.95->boto3->pytorch-pretrained-bert==0.4.0) (2.8.1)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.21.0,>=1.20.95->boto3->pytorch-pretrained-bert==0.4.0) (1.15.0)\n", + "\u001b[31mERROR: botocore 1.20.95 has requirement urllib3<1.27,>=1.25.4, but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n", + "Installing collected packages: jmespath, botocore, s3transfer, boto3, pytorch-pretrained-bert\n", + "Successfully installed boto3-1.17.95 botocore-1.20.95 jmespath-0.10.0 pytorch-pretrained-bert-0.4.0 s3transfer-0.4.2\n", "Collecting seqeval==0.0.12\n", " Downloading https://files.pythonhosted.org/packages/34/91/068aca8d60ce56dd9ba4506850e876aba5e66a6f2f29aa223224b50df0de/seqeval-0.0.12.tar.gz\n", - "Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.6/dist-packages (from seqeval==0.0.12) (1.18.4)\n", - "Requirement already satisfied: Keras>=2.2.4 in /usr/local/lib/python3.6/dist-packages (from seqeval==0.0.12) (2.3.1)\n", - "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (3.13)\n", - "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.4.1)\n", - "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.1.2)\n", - "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.0.8)\n", - "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (2.10.0)\n", - "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.12.0)\n", + "Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from seqeval==0.0.12) (1.19.5)\n", + "Requirement already satisfied: Keras>=2.2.4 in /usr/local/lib/python3.7/dist-packages (from seqeval==0.0.12) (2.4.3)\n", + "Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (3.1.0)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (3.13)\n", + "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.4.1)\n", + "Requirement already satisfied: cached-property; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from h5py->Keras>=2.2.4->seqeval==0.0.12) (1.5.2)\n", "Building wheels for collected packages: seqeval\n", " Building wheel for seqeval (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for seqeval: filename=seqeval-0.0.12-cp36-none-any.whl size=7424 sha256=595ff0deb68dbfc3a88b2ca903c0405d1fa8133d949947c678f02f8432fb3321\n", + " Created wheel for seqeval: filename=seqeval-0.0.12-cp37-none-any.whl size=7436 sha256=35156045a64fe951130c071b95f90a53d025875d52138112f41693fa7fc81b9f\n", " Stored in directory: /root/.cache/pip/wheels/4f/32/0a/df3b340a82583566975377d65e724895b3fad101a3fb729f68\n", "Successfully built seqeval\n", "Installing collected packages: seqeval\n", "Successfully installed seqeval-0.0.12\n", "Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.\n" ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Using TensorFlow backend.\n" - ] } ], "source": [ - "#Installing required packages \n", - "%tensorflow_version 1.x\n", + "#Installing required packages\n", + "try :\n", + " from google.colab import files\n", + " %tensorflow_version 1.x\n", + " \n", + "except ModuleNotFoundError :\n", + " Print(\"Not Using Colab\")\n", + " \n", "!pip install pytorch-pretrained-bert==0.4.0\n", "!pip install seqeval==0.0.12\n", "\n", @@ -90,6 +95,9 @@ "import numpy as np\n", "from tqdm import tqdm, trange\n", "\n", + "import warnings\n", + "warnings.filterwarnings(\"ignore\")\n", + "\n", "#importing all the pytorch packages\n", "import torch\n", "from torch.optim import Adam\n", @@ -108,14 +116,14 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 119, + "height": 106, "resources": { "http://localhost:8080/nbextensions/google.colab/files.js": { - "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=", + "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK", "headers": [ [ "content-type", @@ -124,21 +132,21 @@ ], "ok": true, "status": 200, - "status_text": "OK" + "status_text": "" } } }, - "colab_type": "code", "id": "hVfxrl-HOkTn", - "outputId": "3fb0e72c-43eb-4531-b6b2-bb82de2044c1" + "outputId": "a4770c6a-b778-4233-a733-76f0326eef4e" }, "outputs": [ { "data": { "text/html": [ "\n", - " \n", - " \n", + " \n", + " \n", " Upload widget is only available when the cell has been executed in the\n", " current browser session. Please rerun this cell to enable.\n", " \n", @@ -157,25 +165,26 @@ "name": "stdout", "output_type": "stream", "text": [ - "Saving train.txt to train.txt\n", - "Saving test.txt to test.txt\n" + "Saving test.txt to test.txt\n", + "Saving train.txt to train.txt\n" ] } ], "source": [ "#uploading data into google colab\n", "#upload the test.txt and train.txt files respectively\n", - "from google.colab import files\n", - "uploaded = files.upload()\n", - "\n" + "try :\n", + " from google.colab import files\n", + " uploaded = files.upload()\n", + "\n", + "except ModuleNotFoundError :\n", + " print(\"Not using Colab\")" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 3, "metadata": { - "colab": {}, - "colab_type": "code", "id": "HkbURKk5y0_1" }, "outputs": [], @@ -260,26 +269,41 @@ " wordfeats[\"nextNextTag\"] = sen_tags[i + 2][1]\n", " #That is it! You can add whatever you want!\n", " feats.append(wordfeats)\n", - " return feats\n", - "\n", + " return feats\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "uuURLq19f5fz" + }, + "outputs": [], + "source": [ "#preprocess the data by calling the functions\n", - "train_path = 'train.txt'\n", - "test_path = 'test.txt'\n", + "try :\n", + " from google.colab import files\n", + " train_path = 'train.txt'\n", + " test_path = 'test.txt' \n", + "\n", + "except ModuleNotFoundError :\n", + " train_path = 'Data/conlldata/train.txt'\n", + " test_path = 'Data/conlldata/test.txt'\n", + "\n", + " \n", "conll_train = load__data_conll(train_path)\n", "conll_test = load__data_conll(test_path) " ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 5, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "oYm4B66pBN6T", - "outputId": "1e58c2d2-5871-41d5-9b30-94ee680843e5" + "outputId": "ba0447dc-fcf2-4a3d-93d1-a54de1d3686e" }, "outputs": [ { @@ -293,16 +317,14 @@ "source": [ "#BERT needs us to pre-process the data in a particular way.\n", "#Lets take the raw data from the txt files\n", - "df_train = pd.read_csv(\"train.txt\", engine=\"python\",delimiter=\"\\t\",header=None,encoding='utf-8',error_bad_lines=False)\n", - "df_test = pd.read_csv(\"test.txt\", engine=\"python\",delimiter=\"\\t\",encoding='utf-8',header=None, error_bad_lines=False)" + "df_train = pd.read_csv(train_path, engine=\"python\",delimiter=\"\\t\",header=None,encoding='utf-8',error_bad_lines=False)\n", + "df_test = pd.read_csv(test_path, engine=\"python\",delimiter=\"\\t\",encoding='utf-8',header=None, error_bad_lines=False)" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 6, "metadata": { - "colab": {}, - "colab_type": "code", "id": "tQS3WdwLBOJe" }, "outputs": [], @@ -314,24 +336,30 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "av7yRSvrX4-z", - "outputId": "47275306-9484-4448-f3e8-9ed569eba8dc" + "outputId": "e400cdc3-b2f6-4168-a1ba-556d50b1c4a2" }, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n", + " \"\"\"Entry point for launching an IPython kernel.\n" + ] + }, { "data": { "text/plain": [ "(14041, 2)" ] }, - "execution_count": 8, + "execution_count": 7, "metadata": { "tags": [] }, @@ -344,24 +372,30 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 8, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "3BGxbUspYAPU", - "outputId": "cfe79e7e-8a3c-466e-83be-f53afbd3eb58" + "outputId": "bbff2664-9f50-4464-bfe4-ec4a1b2fcc8e" }, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n", + " \"\"\"Entry point for launching an IPython kernel.\n" + ] + }, { "data": { "text/plain": [ "(3453, 2)" ] }, - "execution_count": 9, + "execution_count": 8, "metadata": { "tags": [] }, @@ -375,7 +409,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "hBA0ccq2YUGN" }, "source": [ @@ -384,10 +417,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 9, "metadata": { - "colab": {}, - "colab_type": "code", "id": "3046zKhaTXOW" }, "outputs": [], @@ -414,7 +445,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "nvtQJ1Fxa89j" }, "source": [ @@ -423,10 +453,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 10, "metadata": { - "colab": {}, - "colab_type": "code", "id": "uEM8-XuPYeWt" }, "outputs": [], @@ -438,15 +466,13 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 11, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 52 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "5WgHCbX_ZAqG", - "outputId": "c524e008-539e-41b6-f5de-e4138e50e521" + "outputId": "f07ba787-8de7-4896-c9e3-cc15806503d5" }, "outputs": [ { @@ -468,24 +494,26 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 34 + "height": 35 }, - "colab_type": "code", "id": "_KW3zKy7ZAnB", - "outputId": "06e86e06-d4e5-4bf6-f184-fde6ec0e8445" + "outputId": "4f63698f-b803-4f3a-e07c-61c8c9d60174" }, "outputs": [ { "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + }, "text/plain": [ "'EU rejects German call to boycott British lamb.'" ] }, - "execution_count": 13, + "execution_count": 12, "metadata": { "tags": [] }, @@ -500,7 +528,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "W88K8YVXh7QA" }, "source": [ @@ -509,17 +536,22 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 13, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "3JUPhdRFzea1", - "outputId": "ba043a5a-62af-4dd6-fa6e-4d1f73e55725" + "outputId": "ecf25f59-5254-4fe0-be36-9ab4e7678b76" }, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 231508/231508 [00:00<00:00, 938747.37B/s]\n" + ] + }, { "name": "stdout", "output_type": "stream", @@ -547,15 +579,13 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 14, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "Niy1e_tU7F0c", - "outputId": "20469205-409c-42fb-cabf-c36b422a6d81" + "outputId": "69913e29-6997-477e-a105-60008eea2ba6" }, "outputs": [ { @@ -564,7 +594,7 @@ "['B-ORG', 'O', 'B-MISC', 'O', 'O', 'O', 'B-MISC', 'O', 'O']" ] }, - "execution_count": 17, + "execution_count": 14, "metadata": { "tags": [] }, @@ -577,10 +607,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 15, "metadata": { - "colab": {}, - "colab_type": "code", "id": "7xxx7mF28NZi" }, "outputs": [], @@ -593,10 +621,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 16, "metadata": { - "colab": {}, - "colab_type": "code", "id": "mvUnUjvk8sOB" }, "outputs": [], @@ -614,10 +640,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 17, "metadata": { - "colab": {}, - "colab_type": "code", "id": "tDb-q3HXEumu" }, "outputs": [], @@ -632,7 +656,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "drV54DRsbYPc" }, "source": [ @@ -641,10 +664,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 18, "metadata": { - "colab": {}, - "colab_type": "code", "id": "_LuJ9LeDEwex" }, "outputs": [], @@ -658,10 +679,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 19, "metadata": { - "colab": {}, - "colab_type": "code", "id": "KuS8WeQyEyG6" }, "outputs": [], @@ -678,15 +697,13 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 20, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 52 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "u-YtGlTXEztA", - "outputId": "9d85ec67-ded0-4b82-fcc6-a432d96ef2e6" + "outputId": "17b852e0-f9eb-4d36-8488-1847ee3f25bc" }, "outputs": [ { @@ -714,22 +731,20 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 21, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 52 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "Ajuboku7E1Y0", - "outputId": "a92f0c7a-8e4f-4b8d-cf1e-559f0563e75a" + "outputId": "0dc44313-ce6b-421c-e278-16a6f5bf8bb1" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████| 407873900/407873900 [00:14<00:00, 28000433.90B/s]\n" + "100%|██████████| 407873900/407873900 [00:09<00:00, 41262134.79B/s]\n" ] }, { @@ -748,343 +763,332 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 23, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "En7S5Jj2E3Sh", - "outputId": "80755596-b372-4977-d68e-d9988615fa91" + "outputId": "c85187b5-f19d-40f9-8a80-82a08be316c8" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Passing Model parameters in GPU\n" + "Passing Model parameters in GPU\n", + "BertForTokenClassification(\n", + " (bert): BertModel(\n", + " (embeddings): BertEmbeddings(\n", + " (word_embeddings): Embedding(30522, 768)\n", + " (position_embeddings): Embedding(512, 768)\n", + " (token_type_embeddings): Embedding(2, 768)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (encoder): BertEncoder(\n", + " (layer): ModuleList(\n", + " (0): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (1): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (2): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (3): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (4): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (5): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (6): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (7): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (8): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (9): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (10): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (11): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " )\n", + " )\n", + " (pooler): BertPooler(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (activation): Tanh()\n", + " )\n", + " )\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (classifier): Linear(in_features=768, out_features=9, bias=True)\n", + ")\n" ] - }, - { - "data": { - "text/plain": [ - "BertForTokenClassification(\n", - " (bert): BertModel(\n", - " (embeddings): BertEmbeddings(\n", - " (word_embeddings): Embedding(30522, 768)\n", - " (position_embeddings): Embedding(512, 768)\n", - " (token_type_embeddings): Embedding(2, 768)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (encoder): BertEncoder(\n", - " (layer): ModuleList(\n", - " (0): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (1): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (2): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (3): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (4): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (5): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (6): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (7): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (8): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (9): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (10): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (11): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " )\n", - " )\n", - " (pooler): BertPooler(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (activation): Tanh()\n", - " )\n", - " )\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " (classifier): Linear(in_features=768, out_features=9, bias=True)\n", - ")" - ] - }, - "execution_count": 25, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" } ], "source": [ "#Passing model parameters into GPU\n", - "print(\"Passing Model parameters in GPU\")\n", - "model.cuda()" + "if torch.cuda.is_available(): \n", + " print(\"Passing Model parameters in GPU\")\n", + " print(model.cuda()) \n", + "else: \n", + " print(model)" ] }, { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "3dlWDBc1az7H" }, "source": [ @@ -1093,15 +1097,13 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 24, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "it5l2g-cE439", - "outputId": "9bd08b85-c7b3-42bf-e1c3-c9a58545e9a4" + "outputId": "2f494a2a-93ff-4414-87ee-275e5459b068" }, "outputs": [ { @@ -1134,10 +1136,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 25, "metadata": { - "colab": {}, - "colab_type": "code", "id": "w7jNHHs1E6u1" }, "outputs": [], @@ -1151,106 +1151,98 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 26, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 486 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "nZHdynvRFDv1", - "outputId": "dcacc02d-de28-4275-a549-c5cd440c6bca" + "outputId": "3138d86b-6eae-4b00-b196-f6d5e307a8e6" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "\n", - "\n", - "Epoch: 0%| | 0/4 [00:00" ] @@ -1361,30 +1352,29 @@ "plt.title(\"Training loss\")\n", "plt.xlabel(\"Batch\")\n", "plt.ylabel(\"Loss\")\n", + "plt.ylim(0,0.25)\n", "plt.plot(train_loss_set)\n", "plt.show()" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 32, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 69 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "Wqic4K83FLKX", - "outputId": "d2c2f4a4-9813-41c5-ba80-2f23e05f4613" + "outputId": "6b419548-c141-4cb0-fe46-0eb912c719ae" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Validation loss: 0.030850813669067893\n", - "Validation Accuracy: 0.9891046831955919\n", - "Validation F1-Score: 0.7566073587839004\n" + "Validation loss: 0.029934273833747613\n", + "Validation Accuracy: 0.9896167355371898\n", + "Validation F1-Score: 0.7633905356214249\n" ] } ], @@ -1423,32 +1413,13 @@ "print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n", "print(\"Validation F1-Score: {}\".format(f1_score(pred_tags, valid_tags)))" ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "gpo5b-hOhR5l" - }, - "source": [] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "9KV7O67dcoJY" - }, - "outputs": [], - "source": [] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], - "name": "BERT_CONLL_NER.ipynb", + "name": "05_BERT_CONLL_NER.ipynb", "provenance": [] }, "kernelspec": { @@ -1466,7 +1437,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.10" + "version": "3.6.13" } }, "nbformat": 4, diff --git a/Ch6/02_BERT_ATIS.ipynb b/Ch6/02_BERT_ATIS.ipynb index 4e45a20..1a64e7c 100644 --- a/Ch6/02_BERT_ATIS.ipynb +++ b/Ch6/02_BERT_ATIS.ipynb @@ -3,7 +3,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "-m92wDoNlzcg" }, "source": [ @@ -15,7 +14,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "EVr-bWUpVUBC" }, "source": [ @@ -25,15 +23,13 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 1, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 357 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", - "id": "d-mlOxsO-cgl", - "outputId": "5834cd19-fa44-424a-b82a-375048e03e66" + "id": "3XHt3WCThUS-", + "outputId": "48f904e0-f918-43fe-9491-0f63c38636ef" }, "outputs": [ { @@ -42,38 +38,29 @@ "text": [ "\u001b[31mERROR: Invalid requirement: 'pytorch-pretrained-bert=0.4.0'\n", "Hint: = is not a valid operator. Did you mean == ?\u001b[0m\n", - "Collecting seqeval==0.0.12\n", - " Downloading https://files.pythonhosted.org/packages/34/91/068aca8d60ce56dd9ba4506850e876aba5e66a6f2f29aa223224b50df0de/seqeval-0.0.12.tar.gz\n", - "Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.6/dist-packages (from seqeval==0.0.12) (1.18.4)\n", - "Requirement already satisfied: Keras>=2.2.4 in /usr/local/lib/python3.6/dist-packages (from seqeval==0.0.12) (2.3.1)\n", - "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.4.1)\n", - "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.0.8)\n", - "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.12.0)\n", - "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (2.10.0)\n", - "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.1.2)\n", - "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (3.13)\n", - "Building wheels for collected packages: seqeval\n", - " Building wheel for seqeval (setup.py) ... \u001b[?25l\u001b[?25hdone\n", - " Created wheel for seqeval: filename=seqeval-0.0.12-cp36-none-any.whl size=7424 sha256=46613f6d831c1fbc3a2df303488acf9bf60dd2dbc39c031e7890dd5c8b78f3c5\n", - " Stored in directory: /root/.cache/pip/wheels/4f/32/0a/df3b340a82583566975377d65e724895b3fad101a3fb729f68\n", - "Successfully built seqeval\n", - "Installing collected packages: seqeval\n", - "Successfully installed seqeval-0.0.12\n" + "Requirement already satisfied: seqeval==0.0.12 in /usr/local/lib/python3.7/dist-packages (0.0.12)\n", + "Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from seqeval==0.0.12) (1.19.5)\n", + "Requirement already satisfied: Keras>=2.2.4 in /usr/local/lib/python3.7/dist-packages (from seqeval==0.0.12) (2.4.3)\n", + "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.4.1)\n", + "Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (3.1.0)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (3.13)\n", + "Requirement already satisfied: cached-property; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from h5py->Keras>=2.2.4->seqeval==0.0.12) (1.5.2)\n" ] - }, - { - "data": { - "text/plain": [ - "'Tesla P100-PCIE-16GB'" - ] - }, - "execution_count": 21, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" } ], + "source": [ + "# install\n", + "!pip install pytorch-pretrained-bert=0.4.0 pytorch-nlp\n", + "!pip install seqeval==0.0.12" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "id": "d-mlOxsO-cgl" + }, + "outputs": [], "source": [ "#importing a few necessary packages and setting the DATA directory\n", "DATA_DIR=\".\"\n", @@ -83,10 +70,6 @@ "import tensorflow as tf\n", "\n", "\n", - "# install\n", - "!pip install pytorch-pretrained-bert=0.4.0 pytorch-nlp\n", - "!pip install seqeval==0.0.12\n", - "\n", "# BERT imports\n", "import torch\n", "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n", @@ -100,21 +83,21 @@ "import io\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", - "% matplotlib inline\n", + "%matplotlib inline\n", "\n", "#importing packages to calculate the f1_score of our model\n", "from seqeval.metrics import f1_score\n", "\n", "# specify GPU device\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", - "n_gpu = torch.cuda.device_count()\n", - "torch.cuda.get_device_name(0)" + "if torch.cuda.is_available():\n", + " n_gpu = torch.cuda.device_count()\n", + " torch.cuda.get_device_name(0)" ] }, { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "M1D9QaVXVejl" }, "source": [ @@ -124,14 +107,14 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 176, + "height": 106, "resources": { "http://localhost:8080/nbextensions/google.colab/files.js": { - "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=", + "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK", "headers": [ [ "content-type", @@ -144,17 +127,17 @@ } } }, - "colab_type": "code", "id": "LPzhwGqK-kXc", - "outputId": "69a03537-2bd4-4e2f-ede6-931b785e8a3c" + "outputId": "3eb1aef8-f48b-49d8-d1ce-9684d2769847" }, "outputs": [ { "data": { "text/html": [ "\n", - " \n", - " \n", + " \n", + " \n", " Upload widget is only available when the cell has been executed in the\n", " current browser session. Please rerun this cell to enable.\n", " \n", @@ -173,24 +156,24 @@ "name": "stdout", "output_type": "stream", "text": [ - "Saving atis.train.w-intent.iob to atis.train.w-intent.iob\n", "Saving atis.test.w-intent.iob to atis.test.w-intent.iob\n", - "Saving atis-2.train.w-intent.iob to atis-2.train.w-intent.iob\n", - "Saving atis-2.dev.w-intent.iob to atis-2.dev.w-intent.iob\n" + "Saving atis.train.w-intent.iob to atis.train.w-intent.iob\n" ] } ], "source": [ "#importing the dataset\n", - "#upload the \"atis.test.w-intent.iob\" and atis.train.w-intent.iob\" \n", - "from google.colab import files\n", - "uploaded = files.upload()" + "#upload the \"atis.test.w-intent.iob\" and atis.train.w-intent.iob\" present in data2 folder\n", + "try:\n", + " from google.colab import files\n", + " uploaded = files.upload()\n", + "except ModuleNotFoundError:\n", + " print('Not using colab')" ] }, { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "6OQLsmbqVqfN" }, "source": [ @@ -200,10 +183,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 4, "metadata": { - "colab": {}, - "colab_type": "code", "id": "Dg16o9IO_BDd" }, "outputs": [], @@ -228,23 +209,12 @@ " sents.append(words[1:-1])\n", " labels.append(labs[1:-1])\n", " intents.append(labs[-1])\n", - " return np.array(sents),np.array(labels),np.array(intents)\n", - "\n", - "read_method = {'atis-2.dev.w-intent.iob':get_data,\n", - " 'atis.train.w-intent.iob':get_data2,\n", - " 'atis.test.w-intent.iob':get_data,\n", - " 'atis-2.train.w-intent.iob':get_data2}\n", - "\n", - "def fetch_data(fname):\n", - " func = read_method[fname]\n", - " return func(fname)\n", - "\n" + " return np.array(sents),np.array(labels),np.array(intents)" ] }, { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "m-ND3usZV8UO" }, "source": [ @@ -253,15 +223,13 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 5, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 153 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "Xc4ULfDU_T16", - "outputId": "ad31b7c0-015f-4a52-9a15-d15c9db101a0" + "outputId": "ca9aaea6-7e74-49be-8cfd-5cae86d08480" }, "outputs": [ { @@ -277,11 +245,23 @@ "('cheapest airfare from tacoma to orlando', 'atis_airfare')\n", "('round trip fares from pittsburgh to philadelphia under 1000 dollars', 'atis_airfare')\n" ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:21: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n" + ] } ], "source": [ "print(\"Loading the training Data\")\n", - "sents,labels,intents = fetch_data('atis.train.w-intent.iob')\n", + "\n", + "try:\n", + " from google.colab import files\n", + " sents,labels,intents = get_data2('atis.train.w-intent.iob')\n", + "except ModuleNotFoundError:\n", + " sents,labels,intents = get_data2('Data/data2/atis.train.w-intent.iob')\n", "\n", "train_sentences = [\" \".join(i) for i in sents]\n", "\n", @@ -308,7 +288,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "GsJTkImyV_r8" }, "source": [ @@ -317,15 +296,13 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 6, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 170 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "tWJ8th5TBsrW", - "outputId": "ac993860-92a5-48d0-d07b-4aa301f37855" + "outputId": "3f20c996-214c-41ea-98d5-9d8edfca3969" }, "outputs": [ { @@ -342,10 +319,23 @@ "('i would like a flight traveling one way from phoenix to san diego on april first', 'atis_flight')\n", "('i would like a flight from orlando to salt lake city for april first on delta airlines', 'atis_flight')\n" ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:10: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n", + " # Remove the CWD from sys.path while we load stuff.\n" + ] } ], "source": [ - "sents,labels,intents = fetch_data('atis.test.w-intent.iob')\n", + "try:\n", + " from google.colab import files\n", + " sents,labels,intents = get_data('atis.test.w-intent.iob')\n", + "except ModuleNotFoundError:\n", + " sents,labels,intents = get_data('Data/data2/atis.test.w-intent.iob')\n", + "\n", "\n", "test_sentences = [\" \".join(i) for i in sents]\n", "\n", @@ -377,7 +367,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "Tymf0cdgWCy5" }, "source": [ @@ -387,15 +376,13 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 7, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "jpbJHj6GXM3o", - "outputId": "deaf59d0-6874-4a8e-d963-baaa66964d46" + "outputId": "9fd10b0e-92c2-4606-cb56-6c6c431af512" }, "outputs": [ { @@ -408,7 +395,6 @@ } ], "source": [ - "\n", "from sklearn.preprocessing import LabelEncoder\n", "\n", "df = pd.DataFrame(data =zip(train_labels,train_texts),columns=['Labels',\"Text\"])\n", @@ -425,7 +411,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "WrmcXpsyWqeW" }, "source": [ @@ -434,44 +419,20 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 8, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 85 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "wKZgpzwLB5jD", - "outputId": "b45b2846-01c4-4337-c6cc-79cbd9b17295" + "outputId": "7eb9a8ad-2237-4335-ea3c-ca184acd5c6e" }, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\r", - " 0%| | 0/231508 [00:000) for i in seq]\n", - " attention_masks.append(seq_mask)\n", + " seq_mask = [float(i>0) for i in seq]\n", + " attention_masks.append(seq_mask)\n", " " ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 11, "metadata": { - "colab": {}, - "colab_type": "code", "id": "46r7l4GZgFTE" }, "outputs": [], @@ -556,7 +510,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "qUv9j3v1aKYg" }, "source": [ @@ -565,10 +518,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 12, "metadata": { - "colab": {}, - "colab_type": "code", "id": "LFsyID-US8s8" }, "outputs": [], @@ -608,15 +559,13 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 13, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "nEHUwpWSTGy0", - "outputId": "87a355e9-0b73-4c80-cb68-3323945ec818" + "outputId": "898bc23f-eb37-4c03-a95b-e15a64046b3b" }, "outputs": [ { @@ -921,7 +870,7 @@ ")" ] }, - "execution_count": 17, + "execution_count": 13, "metadata": { "tags": [] }, @@ -930,13 +879,15 @@ ], "source": [ "model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=17)\n", - "model.cuda()\n" + "if torch.cuda.is_available():\n", + " model.cuda()\n", + "else:\n", + " model" ] }, { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "CgcfFlczaW_V" }, "source": [ @@ -945,15 +896,14 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 836 + "height": 788 }, - "colab_type": "code", "id": "MZFkgV9ybATM", - "outputId": "d6f53a18-831d-4de7-cef5-b0d8ecd06523" + "outputId": "897ea6d8-afdf-47dd-db6d-b72313c9c878" }, "outputs": [ { @@ -961,81 +911,80 @@ "output_type": "stream", "text": [ "t_total value of -1 results in schedule not being applied\n", - "\n", - "Epoch: 0%| | 0/4 [00:00" ] @@ -1063,9 +1012,9 @@ "# BERT fine-tuning parameters\n", "FULL_FINETUNING = True\n", "if FULL_FINETUNING:\n", - " param_optimizer = list(model.named_parameters())\n", - " no_decay = ['bias', 'gamma', 'beta']\n", - " optimizer_grouped_parameters = [\n", + " param_optimizer = list(model.named_parameters())\n", + " no_decay = ['bias', 'gamma', 'beta']\n", + " optimizer_grouped_parameters = [\n", " {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n", " 'weight_decay_rate': 0.01},\n", " {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n", @@ -1094,56 +1043,56 @@ " \n", " ###### TRAINING ######\n", " \n", - " # Set our model to training mode\n", - " model.train() \n", - " # Tracking variables\n", - " tr_loss = 0\n", - " nb_tr_examples, nb_tr_steps = 0, 0\n", - " # Train the data for one epoch\n", - " for step, batch in enumerate(train_dataloader):\n", - " # Add batch to GPU\n", - " batch = tuple(t.to(device) for t in batch)\n", - " # Unpack the inputs from our dataloader\n", - " b_input_ids, b_input_mask, b_labels = batch\n", - " # Clear out the gradients (by default they accumulate)\n", - " optimizer.zero_grad()\n", - " # Forward pass\n", - " loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n", - " train_loss_set.append(loss.item()) \n", - " # Backward pass\n", - " loss.backward()\n", - " # Update parameters and take a step using the computed gradient\n", - " optimizer.step()\n", - " # Update tracking variables\n", - " tr_loss += loss.item()\n", - " nb_tr_examples += b_input_ids.size(0)\n", - " nb_tr_steps += 1\n", - " print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n", + " # Set our model to training mode\n", + " model.train() \n", + " # Tracking variables\n", + " tr_loss = 0\n", + " nb_tr_examples, nb_tr_steps = 0, 0\n", + " # Train the data for one epoch\n", + " for step, batch in enumerate(train_dataloader):\n", + " # Add batch to GPU\n", + " batch = tuple(t.to(device) for t in batch)\n", + " # Unpack the inputs from our dataloader\n", + " b_input_ids, b_input_mask, b_labels = batch\n", + " # Clear out the gradients (by default they accumulate)\n", + " optimizer.zero_grad()\n", + " # Forward pass\n", + " loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n", + " train_loss_set.append(loss.item()) \n", + " # Backward pass\n", + " loss.backward()\n", + " # Update parameters and take a step using the computed gradient\n", + " optimizer.step()\n", + " # Update tracking variables\n", + " tr_loss += loss.item()\n", + " nb_tr_examples += b_input_ids.size(0)\n", + " nb_tr_steps += 1\n", + " print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n", " \n", " ###### VALIDATION ######\n", "\n", " # Put model in evaluation mode\n", - " model.eval()\n", - " # Tracking variables \n", - " eval_loss, eval_accuracy = 0, 0\n", - " nb_eval_steps, nb_eval_examples = 0, 0\n", - " # Evaluate data for one epoch\n", - " for batch in validation_dataloader:\n", - " # Add batch to GPU\n", - " batch = tuple(t.to(device) for t in batch)\n", - " # Unpack the inputs from our dataloader\n", - " b_input_ids, b_input_mask, b_labels = batch\n", - " # Telling the model not to compute or store gradients, saving memory and speeding up validation\n", - " with torch.no_grad():\n", - " # Forward pass, calculate logit predictions\n", - " logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask) \n", - " # Move logits and labels to CPU\n", - " logits = logits.detach().cpu().numpy()\n", - " label_ids = b_labels.to('cpu').numpy()\n", - " tmp_eval_accuracy = flat_accuracy(logits, label_ids) \n", - " eval_accuracy += tmp_eval_accuracy\n", - " nb_eval_steps += 1\n", - " print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n", + " model.eval()\n", + " # Tracking variables \n", + " eval_loss, eval_accuracy = 0, 0\n", + " nb_eval_steps, nb_eval_examples = 0, 0\n", + " # Evaluate data for one epoch\n", + " for batch in validation_dataloader:\n", + " # Add batch to GPU\n", + " batch = tuple(t.to(device) for t in batch)\n", + " # Unpack the inputs from our dataloader\n", + " b_input_ids, b_input_mask, b_labels = batch\n", + " # Telling the model not to compute or store gradients, saving memory and speeding up validation\n", + " with torch.no_grad():\n", + " # Forward pass, calculate logit predictions\n", + " logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask) \n", + " # Move logits and labels to CPU\n", + " logits = logits.detach().cpu().numpy()\n", + " label_ids = b_labels.to('cpu').numpy()\n", + " tmp_eval_accuracy = flat_accuracy(logits, label_ids) \n", + " eval_accuracy += tmp_eval_accuracy\n", + " nb_eval_steps += 1\n", + " print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n", "\n", "# plot training performance\n", "plt.figure(figsize=(15,8))\n", @@ -1158,7 +1107,7 @@ "metadata": { "accelerator": "GPU", "colab": { - "name": "BERT_ATIS_NEW.ipynb", + "name": "02_BERT_ATIS.ipynb", "provenance": [] }, "kernelspec": { @@ -1176,7 +1125,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.12" + "version": "3.7.4" } }, "nbformat": 4, diff --git a/Ch6/03_BERT_ATIS_Binary.ipynb b/Ch6/03_BERT_ATIS_Binary.ipynb index a870309..25d3d4c 100644 --- a/Ch6/03_BERT_ATIS_Binary.ipynb +++ b/Ch6/03_BERT_ATIS_Binary.ipynb @@ -3,7 +3,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "b7We9WSKH7DN" }, "source": [ @@ -13,92 +12,61 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "id": "vKbMCwXXgqC9" + }, "source": [ "## Imports" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 1, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 546 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "Mk2-vK00E2ms", - "outputId": "beab72e6-87de-4c58-9a56-00e371139a0e" + "outputId": "e2dec994-33b8-46fe-a0bd-0f09ed0ee6ec" }, "outputs": [ - { - "data": { - "text/html": [ - "

\n", - "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.
\n", - "We recommend you upgrade now \n", - "or ensure your notebook will continue to use TensorFlow 1.x via the %tensorflow_version 1.x magic:\n", - "more info.

\n" - ], - "text/plain": [ - "" - ] - }, - "metadata": { - "tags": [] - }, - "output_type": "display_data" - }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting pytorch-pretrained-bert\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d7/e0/c08d5553b89973d9a240605b9c12404bcf8227590de62bae27acbcfe076b/pytorch_pretrained_bert-0.6.2-py3-none-any.whl (123kB)\n", - "\r", - "\u001b[K |██▋ | 10kB 22.7MB/s eta 0:00:01\r", - "\u001b[K |█████▎ | 20kB 2.1MB/s eta 0:00:01\r", - "\u001b[K |████████ | 30kB 3.1MB/s eta 0:00:01\r", - "\u001b[K |██████████▋ | 40kB 2.1MB/s eta 0:00:01\r", - "\u001b[K |█████████████▎ | 51kB 2.5MB/s eta 0:00:01\r", - "\u001b[K |███████████████▉ | 61kB 3.0MB/s eta 0:00:01\r", - "\u001b[K |██████████████████▌ | 71kB 3.5MB/s eta 0:00:01\r", - "\u001b[K |█████████████████████▏ | 81kB 3.9MB/s eta 0:00:01\r", - "\u001b[K |███████████████████████▉ | 92kB 4.4MB/s eta 0:00:01\r", - "\u001b[K |██████████████████████████▌ | 102kB 3.4MB/s eta 0:00:01\r", - "\u001b[K |█████████████████████████████▏ | 112kB 3.4MB/s eta 0:00:01\r", - "\u001b[K |███████████████████████████████▊| 122kB 3.4MB/s eta 0:00:01\r", - "\u001b[K |████████████████████████████████| 133kB 3.4MB/s \n", + "\u001b[K |████████████████████████████████| 133kB 7.9MB/s \n", "\u001b[?25hCollecting pytorch-nlp\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/4f/51/f0ee1efb75f7cc2e3065c5da1363d6be2eec79691b2821594f3f2329528c/pytorch_nlp-0.5.0-py3-none-any.whl (90kB)\n", - "\r", - "\u001b[K |███▋ | 10kB 26.7MB/s eta 0:00:01\r", - "\u001b[K |███████▎ | 20kB 32.9MB/s eta 0:00:01\r", - "\u001b[K |███████████ | 30kB 39.6MB/s eta 0:00:01\r", - "\u001b[K |██████████████▌ | 40kB 44.0MB/s eta 0:00:01\r", - "\u001b[K |██████████████████▏ | 51kB 46.7MB/s eta 0:00:01\r", - "\u001b[K |█████████████████████▉ | 61kB 48.7MB/s eta 0:00:01\r", - "\u001b[K |█████████████████████████▌ | 71kB 50.1MB/s eta 0:00:01\r", - "\u001b[K |█████████████████████████████ | 81kB 50.9MB/s eta 0:00:01\r", - "\u001b[K |████████████████████████████████| 92kB 12.3MB/s \n", - "\u001b[?25hRequirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (1.10.47)\n", - "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (1.17.5)\n", - "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (4.28.1)\n", - "Requirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (1.3.1)\n", - "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (2.21.0)\n", - "Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (2019.12.20)\n", - "Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert) (0.2.1)\n", - "Requirement already satisfied: botocore<1.14.0,>=1.13.47 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert) (1.13.47)\n", - "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert) (0.9.4)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (2019.11.28)\n", - "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (2.8)\n", - "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (3.0.4)\n", - "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (1.24.3)\n", - "Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= \"2.7\" in /usr/local/lib/python3.6/dist-packages (from botocore<1.14.0,>=1.13.47->boto3->pytorch-pretrained-bert) (2.6.1)\n", - "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.14.0,>=1.13.47->boto3->pytorch-pretrained-bert) (0.15.2)\n", - "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil<3.0.0,>=2.1; python_version >= \"2.7\"->botocore<1.14.0,>=1.13.47->boto3->pytorch-pretrained-bert) (1.12.0)\n", - "Installing collected packages: pytorch-pretrained-bert, pytorch-nlp\n", - "Successfully installed pytorch-nlp-0.5.0 pytorch-pretrained-bert-0.6.2\n" + "\u001b[K |████████████████████████████████| 92kB 10.6MB/s \n", + "\u001b[?25hRequirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert) (1.8.1+cu101)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert) (1.19.5)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert) (2.23.0)\n", + "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert) (4.41.1)\n", + "Collecting boto3\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/7c/e1/1b164502f455035def771ec7a31f705351b7f953695d57ce26219aaf21a9/boto3-1.17.90-py2.py3-none-any.whl (131kB)\n", + "\u001b[K |████████████████████████████████| 133kB 22.1MB/s \n", + "\u001b[?25hRequirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert) (2019.12.20)\n", + "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=0.4.1->pytorch-pretrained-bert) (3.7.4.3)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert) (1.24.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert) (2020.12.5)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert) (3.0.4)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert) (2.10)\n", + "Collecting botocore<1.21.0,>=1.20.90\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/4a/ac/617d3ac25ea905279deb06edd82d6c19ca272006d6dcf232b837b75c3dde/botocore-1.20.90-py2.py3-none-any.whl (7.6MB)\n", + "\u001b[K |████████████████████████████████| 7.6MB 25.6MB/s \n", + "\u001b[?25hCollecting s3transfer<0.5.0,>=0.4.0\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/63/d0/693477c688348654ddc21dcdce0817653a294aa43f41771084c25e7ff9c7/s3transfer-0.4.2-py2.py3-none-any.whl (79kB)\n", + "\u001b[K |████████████████████████████████| 81kB 10.0MB/s \n", + "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n", + " Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.21.0,>=1.20.90->boto3->pytorch-pretrained-bert) (2.8.1)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.21.0,>=1.20.90->boto3->pytorch-pretrained-bert) (1.15.0)\n", + "\u001b[31mERROR: botocore 1.20.90 has requirement urllib3<1.27,>=1.25.4, but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n", + "Installing collected packages: jmespath, botocore, s3transfer, boto3, pytorch-pretrained-bert, pytorch-nlp\n", + "Successfully installed boto3-1.17.90 botocore-1.20.90 jmespath-0.10.0 pytorch-nlp-0.5.0 pytorch-pretrained-bert-0.6.2 s3transfer-0.4.2\n", + "TensorFlow 1.x selected.\n" ] }, { @@ -107,36 +75,28 @@ "text": [ "Using TensorFlow backend.\n" ] - }, - { - "data": { - "text/plain": [ - "'Tesla T4'" - ] - }, - "execution_count": 1, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" } ], "source": [ + "import warnings\n", + "warnings.filterwarnings(\"ignore\")\n", + "\n", + "# install\n", + "!pip install pytorch-pretrained-bert pytorch-nlp\n", "\n", - "#if not using colab, comment below line\n", - "%tensorflow_version 1.x\n", + "try : \n", + " from google.colab import files\n", + " %tensorflow_version 1.x\n", + " \n", + "except ModuleNotFoundError :\n", + " print(\"Not Using Colab\")\n", "\n", - "from torch.nn import Adam\n", "DATA_DIR=\".\"\n", "import os\n", "import numpy as np\n", "import pickle\n", "import tensorflow as tf\n", "\n", - "\n", - "# install\n", - "!pip install pytorch-pretrained-bert pytorch-nlp\n", - "\n", "# BERT imports\n", "import torch\n", "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n", @@ -149,31 +109,34 @@ "import io\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", - "% matplotlib inline\n", + "%matplotlib inline\n", "\n", "# specify GPU device\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", - "n_gpu = torch.cuda.device_count()\n", - "torch.cuda.get_device_name(0)" + "if torch.cuda.is_available():\n", + " n_gpu = torch.cuda.device_count()\n", + " torch.cuda.get_device_name(0)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "id": "xhRHs0sugqC_" + }, "source": [ "## Data Loading" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 109, + "height": 162, "resources": { "http://localhost:8080/nbextensions/google.colab/files.js": { - "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=", + "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK", "headers": [ [ "content-type", @@ -186,17 +149,17 @@ } } }, - "colab_type": "code", "id": "ntUZndfrE9W8", - "outputId": "5798e9c4-0b51-430b-cf75-6b139a55cc10" + "outputId": "fe735e2b-7702-40e8-dbd2-fc7113e78901" }, "outputs": [ { "data": { "text/html": [ "\n", - " \n", - " \n", + " \n", + " \n", " Upload widget is only available when the cell has been executed in the\n", " current browser session. Please rerun this cell to enable.\n", " \n", @@ -218,26 +181,114 @@ "Saving atis.test.pkl to atis.test.pkl\n", "Saving atis.train.pkl to atis.train.pkl\n" ] + }, + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " Upload widget is only available when the cell has been executed in the\n", + " current browser session. Please rerun this cell to enable.\n", + " \n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saving atis.train.w-intent.iob to atis.train.w-intent.iob\n" + ] } ], "source": [ "# importing the dataset\n", - "from google.colab import files\n", - "uploaded = files.upload()\n" + "try :\n", + " from google.colab import files\n", + "# upload 'atis.test.pkl' and 'atis.train.pkl' here\n", + " uploaded = files.upload()\n", + "# upload 'atis.train.w-intent.iob' here\n", + " next_uploaded = files.upload() \n", + " \n", + "except ModuleNotFoundError : \n", + " print(\"Not using Colab\")" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 3, "metadata": { - "colab": {}, - "colab_type": "code", - "id": "2YVAk61inICj" + "id": "T84rxZBGgqC_" }, "outputs": [], + "source": [ + "#preprocessing functions\n", + "def get_data(filename):\n", + " df = pd.read_csv(filename,delim_whitespace=True,names=['word','label'])\n", + " beg_indices = list(df[df['word'] == 'BOS'].index)+[df.shape[0]]\n", + " sents,labels,intents = [],[],[]\n", + " for i in range(len(beg_indices[:-1])):\n", + " sents.append(df[beg_indices[i]+1:beg_indices[i+1]-1]['word'].values)\n", + " labels.append(df[beg_indices[i]+1:beg_indices[i+1]-1]['label'].values)\n", + " intents.append(df.loc[beg_indices[i+1]-1]['label']) \n", + " return np.array(sents),np.array(labels),np.array(intents)\n", + "\n", + "def get_data2(filename):\n", + " with open(filename) as f:\n", + " contents = f.read()\n", + " sents,labels,intents = [],[],[]\n", + " for line in contents.strip().split('\\n'):\n", + " words,labs = [i.split(' ') for i in line.split('\\t')]\n", + " sents.append(words[1:-1])\n", + " labels.append(labs[1:-1])\n", + " intents.append(labs[-1])\n", + " return np.array(sents),np.array(labels),np.array(intents)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "2YVAk61inICj", + "outputId": "ddc00fa9-986e-4a6f-ba56-a3b5b0830ad9" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loading the training Data\n", + "Number of training sentences : 4952\n", + "Number of unique intents : 17\n", + "('i want to fly from boston at 838 am and arrive in denver at 1110 in the morning', 'atis_flight')\n", + "('what flights are available from pittsburgh to baltimore on thursday morning', 'atis_flight')\n", + "('what is the arrival time in san francisco for the 755 am flight leaving washington', 'atis_flight_time')\n", + "('cheapest airfare from tacoma to orlando', 'atis_airfare')\n", + "('round trip fares from pittsburgh to philadelphia under 1000 dollars', 'atis_airfare')\n" + ] + } + ], "source": [ "print(\"Loading the training Data\")\n", - "sents,labels,intents = fetch_data('atis.train.w-intent.iob')\n", + "try : \n", + " from google.colab import files\n", + " sents,labels,intents = get_data2('atis.train.w-intent.iob')\n", + " \n", + "except ModuleNotFoundError :\n", + " sents,labels,intents = get_data2('Data/data2/atis.train.w-intent.iob')\n", "\n", "train_sentences = [\" \".join(i) for i in sents]\n", "\n", @@ -263,15 +314,86 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 5, + "metadata": { + "id": "By1w6cxqgqDB" + }, + "outputs": [], + "source": [ + "# load Pickle file \n", + "def load_ds(fname, verbose=True):\n", + " with open(fname, 'rb') as stream:\n", + " ds,dicts = pickle.load(stream)\n", + " if verbose:\n", + " print('Done loading: ', fname)\n", + " print(' samples: {:4d}'.format(len(ds['query'])))\n", + " print(' vocab_size: {:4d}'.format(len(dicts['token_ids'])))\n", + " print(' slot count: {:4d}'.format(len(dicts['slot_ids'])))\n", + " print(' intent count: {:4d}'.format(len(dicts['intent_ids'])))\n", + " return ds,dicts\n", + " \n", + "# convert Pickle file to arrays\n", + "def load_atis(filename, add_start_end_token=False, verbose=True):\n", + " train_ds, dicts = load_ds(os.path.join(DATA_DIR,filename), verbose)\n", + " t2i, s2i, in2i = map(dicts.get, ['token_ids', 'slot_ids','intent_ids'])\n", + " i2t, i2s, i2in = map(lambda d: {d[k]:k for k in d.keys()}, [t2i,s2i,in2i])\n", + " query, slots, intent = map(train_ds.get, ['query', 'slot_labels', 'intent_labels'])\n", + "\n", + " if add_start_end_token:\n", + " i2s[178] = 'BOS'\n", + " i2s[179] = 'EOS'\n", + " s2i['BOS'] = 178\n", + " s2i['EOS'] = 179\n", + "\n", + " input_tensor = []\n", + " target_tensor = []\n", + " query_data = []\n", + " intent_data = []\n", + " slot_data = []\n", + " to_show = np.random.randint(0, len(query)-1, 5)\n", + " for i in range(len(query)):\n", + " input_tensor.append(query[i])\n", + " slot_text = []\n", + " slot_vector = []\n", + " for j in range(len(query[i])):\n", + " slot_text.append(i2s[slots[i][j]])\n", + " slot_vector.append(slots[i][j])\n", + " if add_start_end_token:\n", + " slot_text[0] = 'BOS'\n", + " slot_vector[0] = 178\n", + " slot_text[-1] = 'EOS'\n", + " slot_vector[-1]= 179\n", + " target_tensor.append(slot_vector)\n", + " q = ' '.join(map(i2t.get, query[i]))\n", + " query_data.append(q.replace('BOS', '').replace('EOS',''))\n", + " intent_data.append(i2in[intent[i][0]])\n", + " slot = ' '.join(slot_text)\n", + " slot_data.append(slot[1:-1])\n", + " if i in to_show and verbose:\n", + " print('Query text:', q)\n", + " print('Query vector: ', query[i])\n", + " print('Intent label: ', i2in[intent[i][0]])\n", + " print('Slot text: ', slot)\n", + " print('Slot vector: ', slot_vector)\n", + " print('*'*74)\n", + " \n", + " query_data = np.array(query_data)\n", + " intent_data = np.array(intent_data)\n", + " slot_data = np.array(slot_data)\n", + " intent_data_label = np.array(intent).flatten()\n", + " \n", + " return t2i, s2i, in2i, i2t, i2s, i2in, input_tensor, target_tensor, query_data, intent_data, intent_data_label, slot_data\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "N4v3gLWxFQS5", - "outputId": "d3269aa2-73a3-4af6-8133-524731b6827f" + "outputId": "c2d734b6-3be6-4f24-d7d0-61e6087a4bff" }, "outputs": [ { @@ -283,100 +405,122 @@ " vocab_size: 943\n", " slot count: 129\n", " intent count: 26\n", - "Query text: BOS please list the flights from charlotte to long beach arriving after lunch time EOS\n", - "Query vector: [178 688 549 827 429 444 294 851 559 252 238 190 569 849 179]\n", - "Intent label: flight\n", - "Slot text: O O O O O O B-fromloc.city_name O B-toloc.city_name I-toloc.city_name O B-arrive_time.time_relative B-arrive_time.period_of_day I-arrive_time.period_of_day O\n", - "Slot vector: [128, 128, 128, 128, 128, 128, 48, 128, 78, 125, 128, 15, 12, 87, 128]\n", + "Query text: BOS show me fares from seattle to minneapolis EOS\n", + "Query vector: [178 770 581 415 444 752 851 597 179]\n", + "Intent label: airfare\n", + "Slot text: O O O O O B-fromloc.city_name O B-toloc.city_name O\n", + "Slot vector: [128, 128, 128, 128, 128, 48, 128, 78, 128]\n", "**************************************************************************\n", - "Query text: BOS sfo to denver on monday november eleventh 1991 EOS\n", - "Query vector: [178 767 851 351 654 601 636 397 61 179]\n", + "Query text: BOS i would like to find a flight from pittsburgh to boston on wednesday and i have to be in boston by one so i 'd like a flight out of here no later than 11 am EOS\n", + "Query vector: [178 479 932 545 851 423 180 428 444 682 851 266 654 908 215 479 463 851\n", + " 250 482 266 277 656 779 479 0 545 180 428 669 646 468 627 531 823 24\n", + " 210 179]\n", "Intent label: flight\n", - "Slot text: O B-fromloc.airport_code O B-toloc.city_name O B-depart_date.day_name B-depart_date.month_name B-depart_date.day_number B-depart_date.year O\n", - "Slot vector: [128, 46, 128, 78, 128, 26, 28, 27, 30, 128]\n", + "Slot text: O O O O O O O O O B-fromloc.city_name O B-toloc.city_name O B-depart_date.day_name O O O O O O B-toloc.city_name B-arrive_time.time_relative B-arrive_time.time O O O O O O O O O B-depart_time.time_relative I-depart_time.time_relative I-depart_time.time_relative O O O\n", + "Slot vector: [128, 128, 128, 128, 128, 128, 128, 128, 128, 48, 128, 78, 128, 26, 128, 128, 128, 128, 128, 128, 78, 15, 14, 128, 128, 128, 128, 128, 128, 128, 128, 128, 36, 101, 101, 128, 128, 128]\n", "**************************************************************************\n", - "Query text: BOS show me all flights from philadelphia to san francisco with one stop in dallas EOS\n", - "Query vector: [178 770 581 207 429 444 678 851 739 440 925 656 797 482 339 179]\n", + "Query text: BOS show me the cheapest flight from pittsburgh to atlanta on wednesday which leaves before noon and serves breakfast EOS\n", + "Query vector: [178 770 581 827 296 428 444 682 851 242 654 908 920 538 253 631 215 758\n", + " 269 179]\n", "Intent label: flight\n", - "Slot text: O O O O O O B-fromloc.city_name O B-toloc.city_name I-toloc.city_name O B-flight_stop I-flight_stop O B-stoploc.city_name O\n", - "Slot vector: [128, 128, 128, 128, 128, 128, 48, 128, 78, 125, 128, 44, 107, 128, 71, 128]\n", + "Slot text: O O O O B-cost_relative O O B-fromloc.city_name O B-toloc.city_name O B-depart_date.day_name O O B-depart_time.time_relative B-depart_time.time O O B-meal_description O\n", + "Slot vector: [128, 128, 128, 128, 21, 128, 128, 48, 128, 78, 128, 26, 128, 128, 36, 35, 128, 128, 53, 128]\n", "**************************************************************************\n", - "Query text: BOS what is the fare on the thrift economy flight from boston to san francisco EOS\n", - "Query vector: [178 916 498 827 414 654 827 843 391 428 444 266 851 739 440 179]\n", - "Intent label: airfare\n", - "Slot text: O O O O O O O B-class_type B-economy O O B-fromloc.city_name O B-toloc.city_name I-toloc.city_name O\n", - "Slot vector: [128, 128, 128, 128, 128, 128, 128, 18, 37, 128, 128, 48, 128, 78, 125, 128]\n", + "Query text: BOS what is the flight number of the earliest flight between boston and washington dc EOS\n", + "Query vector: [178 916 498 827 428 638 646 827 387 428 259 266 215 905 344 179]\n", + "Intent label: flight_no\n", + "Slot text: O O O O O O O O B-flight_mod O O B-fromloc.city_name O B-toloc.city_name B-toloc.state_code O\n", + "Slot vector: [128, 128, 128, 128, 128, 128, 128, 128, 42, 128, 128, 48, 128, 78, 80, 128]\n", "**************************************************************************\n", - "Query text: BOS i would like information on flights from baltimore to atlanta on thursday arriving in atlanta before 6 pm EOS\n", - "Query vector: [178 479 932 545 492 654 429 444 247 851 242 654 845 238 482 242 253 130\n", - " 689 179]\n", - "Intent label: flight\n", - "Slot text: O O O O O O O O B-fromloc.city_name O B-toloc.city_name O B-depart_date.day_name O O B-toloc.city_name B-arrive_time.time_relative B-arrive_time.time I-arrive_time.time O\n", - "Slot vector: [128, 128, 128, 128, 128, 128, 128, 128, 48, 128, 78, 128, 26, 128, 128, 78, 15, 14, 89, 128]\n", + "Query text: BOS what is ff EOS\n", + "Query vector: [178 916 498 417 179]\n", + "Intent label: abbreviation\n", + "Slot text: O O O B-airline_code O\n", + "Slot vector: [128, 128, 128, 1, 128]\n", "**************************************************************************\n", "Done loading: ./atis.test.pkl\n", " samples: 893\n", " vocab_size: 943\n", " slot count: 129\n", " intent count: 26\n", - "Query text: BOS show me the cheapest first class round trip from new york to miami EOS\n", - "Query vector: [178 770 581 827 296 425 302 730 870 444 619 937 851 589 179]\n", + "Query text: BOS which flights leave chicago next tuesday and arrive in detroit around 6 pm EOS\n", + "Query vector: [178 920 429 537 297 621 874 215 236 482 361 231 130 689 179]\n", "Intent label: flight\n", - "Slot text: O O O O B-cost_relative B-class_type I-class_type B-round_trip I-round_trip O B-fromloc.city_name I-fromloc.city_name O B-toloc.city_name O\n", - "Slot vector: [128, 128, 128, 128, 21, 18, 92, 66, 119, 128, 48, 110, 128, 78, 128]\n", + "Slot text: O O O O B-fromloc.city_name B-depart_date.date_relative B-depart_date.day_name O O O B-toloc.city_name B-arrive_time.time_relative B-arrive_time.time I-arrive_time.time O\n", + "Slot vector: [128, 128, 128, 128, 48, 25, 26, 128, 128, 128, 78, 15, 14, 89, 128]\n", "**************************************************************************\n", - "Query text: BOS show me flights between detroit and st. louis on delta northwest us air and united airlines EOS\n", - "Query vector: [178 770 581 429 259 361 215 789 564 654 350 634 889 195 215 887 200 179]\n", + "Query text: BOS please give me the flights from nashville to houston nonstop with dinner served EOS\n", + "Query vector: [178 688 449 581 827 429 444 614 851 476 629 925 367 757 179]\n", "Intent label: flight\n", - "Slot text: O O O O O B-fromloc.city_name O B-toloc.city_name I-toloc.city_name O B-airline_name B-airline_name B-airline_name I-airline_name O B-airline_name I-airline_name O\n", - "Slot vector: [128, 128, 128, 128, 128, 48, 128, 78, 125, 128, 2, 2, 2, 83, 128, 2, 83, 128]\n", + "Slot text: O O O O O O O B-fromloc.city_name O B-toloc.city_name B-flight_stop O B-meal_description O O\n", + "Slot vector: [128, 128, 128, 128, 128, 128, 128, 48, 128, 78, 44, 128, 53, 128, 128]\n", "**************************************************************************\n", - "Query text: BOS give me the flights from milwaukee to phoenix on saturday or sunday on american airlines EOS\n", - "Query vector: [178 449 581 827 429 444 595 851 681 654 740 662 805 654 212 200 179]\n", + "Query text: BOS i want to fly from san francisco to milwaukee and from milwaukee to denver EOS\n", + "Query vector: [178 479 902 851 431 444 739 440 851 595 215 444 595 851 351 179]\n", "Intent label: flight\n", - "Slot text: O O O O O O B-fromloc.city_name O B-toloc.city_name O B-depart_date.day_name B-or B-depart_date.day_name O B-airline_name I-airline_name O\n", - "Slot vector: [128, 128, 128, 128, 128, 128, 48, 128, 78, 128, 26, 56, 26, 128, 2, 83, 128]\n", + "Slot text: O O O O O O B-fromloc.city_name I-fromloc.city_name O B-toloc.city_name O O B-fromloc.city_name O B-toloc.city_name O\n", + "Slot vector: [128, 128, 128, 128, 128, 128, 48, 110, 128, 78, 128, 128, 48, 128, 78, 128]\n", "**************************************************************************\n", - "Query text: BOS please find all the flights from cincinnati to any airport in the new york city area that arrive next saturday before 6 pm EOS\n", - "Query vector: [178 688 423 207 827 429 444 299 851 218 203 482 827 619 937 301 229 826\n", - " 236 621 740 253 130 689 179]\n", + "Query text: BOS what is the earliest arriving flight between boston and washington dc EOS\n", + "Query vector: [178 916 498 827 387 238 428 259 266 215 905 344 179]\n", "Intent label: flight\n", - "Slot text: O O O O O O O B-fromloc.city_name O O O O O B-toloc.city_name I-toloc.city_name I-toloc.city_name O O O B-arrive_date.date_relative B-arrive_date.day_name B-arrive_time.time_relative B-arrive_time.time I-arrive_time.time O\n", - "Slot vector: [128, 128, 128, 128, 128, 128, 128, 48, 128, 128, 128, 128, 128, 78, 125, 125, 128, 128, 128, 5, 6, 15, 14, 89, 128]\n", + "Slot text: O O O O B-flight_mod I-flight_mod O O B-fromloc.city_name O B-toloc.city_name B-toloc.state_code O\n", + "Slot vector: [128, 128, 128, 128, 42, 105, 128, 128, 48, 128, 78, 80, 128]\n", + "**************************************************************************\n", + "Query text: BOS i 'd like a round trip flight from kansas city to chicago on wednesday may twenty sixth arriving at 7 pm EOS\n", + "Query vector: [178 479 0 545 180 730 870 428 444 511 301 851 297 654 908 578 881 775\n", + " 238 240 139 689 179]\n", + "Intent label: flight\n", + "Slot text: O O O O O B-round_trip I-round_trip O O B-fromloc.city_name I-fromloc.city_name O B-toloc.city_name O B-depart_date.day_name B-depart_date.month_name B-depart_date.day_number I-depart_date.day_number O O B-arrive_time.time I-arrive_time.time O\n", + "Slot vector: [128, 128, 128, 128, 128, 66, 119, 128, 128, 48, 110, 128, 78, 128, 26, 28, 27, 95, 128, 128, 14, 89, 128]\n", "**************************************************************************\n" ] } ], "source": [ "# load ATIS training dataset\n", - "t2i_train, s2i_train, in2i_train, i2t_train, i2s_train, i2in_train, \\\n", - "input_tensor_train, target_tensor_train, \\\n", - "query_data_train, intent_data_train, intent_data_label_train, slot_data_train = load_atis('atis.train.pkl')\n", + "try : \n", + " from google.colab import files\n", + " t2i_train, s2i_train, in2i_train, i2t_train, i2s_train, i2in_train, \\\n", + " input_tensor_train, target_tensor_train, \\\n", + " query_data_train, intent_data_train, intent_data_label_train, slot_data_train = load_atis('atis.train.pkl')\n", + " \n", + "except ModuleNotFoundError :\n", + " t2i_train, s2i_train, in2i_train, i2t_train, i2s_train, i2in_train, \\\n", + " input_tensor_train, target_tensor_train, \\\n", + " query_data_train, intent_data_train, intent_data_label_train, slot_data_train = load_atis('Data/data/atis.train.pkl')\n", "\n", "# load ATIS testing dataset\n", - "t2i_test, s2i_test, in2i_test, i2t_test, i2s_test, i2in_test, \\\n", - "input_tensor_test, target_tensor_test, \\\n", - "query_data_test, intent_data_test, intent_data_label_test, slot_data_test = load_atis('atis.test.pkl')\n" + "try : \n", + " from google.colab import files\n", + " t2i_test, s2i_test, in2i_test, i2t_test, i2s_test, i2in_test, \\\n", + " input_tensor_test, target_tensor_test, \\\n", + " query_data_test, intent_data_test, intent_data_label_test, slot_data_test = load_atis('atis.test.pkl')\n", + " \n", + "except ModuleNotFoundError :\n", + " t2i_test, s2i_test, in2i_test, i2t_test, i2s_test, i2in_test, \\\n", + " input_tensor_test, target_tensor_test, \\\n", + " query_data_test, intent_data_test, intent_data_label_test, slot_data_test = load_atis('Data/data/atis.test.pkl')\n" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "id": "JkMdQoGYgqDD" + }, "source": [ "Let's look at a few training queries." ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 7, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 141 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "n9QxatGPpXj2", - "outputId": "350df9a0-b551-42b7-b244-a8530818bd44" + "outputId": "60ecf78b-2c64-4a44-a138-e4e55a099359" }, "outputs": [ { @@ -391,7 +535,7 @@ " dtype='0) for i in seq]\n", - " attention_masks.append(seq_mask)\n", + " seq_mask = [float(i>0) for i in seq]\n", + " attention_masks.append(seq_mask)\n", " " ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 12, "metadata": { - "colab": {}, - "colab_type": "code", "id": "oL39XXZorV0y" }, "outputs": [], @@ -577,10 +717,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 13, "metadata": { - "colab": {}, - "colab_type": "code", "id": "vSV8Kk15pJXL" }, "outputs": [], @@ -613,363 +751,61 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "id": "BuKhVmSFgqDH" + }, "source": [ "## Training" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 14, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "z4Y9kjn7puKI", - "outputId": "07382972-9b50-4c2f-b202-6d000ed98e83" + "outputId": "44656e9d-a314-4c20-b978-ebc63386bd83", + "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████| 407873900/407873900 [00:05<00:00, 74783111.36B/s]\n" + "100%|██████████| 407873900/407873900 [00:11<00:00, 34579258.61B/s]\n" ] - }, - { - "data": { - "text/plain": [ - "BertForSequenceClassification(\n", - " (bert): BertModel(\n", - " (embeddings): BertEmbeddings(\n", - " (word_embeddings): Embedding(30522, 768, padding_idx=0)\n", - " (position_embeddings): Embedding(512, 768)\n", - " (token_type_embeddings): Embedding(2, 768)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (encoder): BertEncoder(\n", - " (layer): ModuleList(\n", - " (0): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (1): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (2): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (3): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (4): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (5): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (6): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (7): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (8): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (9): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (10): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (11): BertLayer(\n", - " (attention): BertAttention(\n", - " (self): BertSelfAttention(\n", - " (query): Linear(in_features=768, out_features=768, bias=True)\n", - " (key): Linear(in_features=768, out_features=768, bias=True)\n", - " (value): Linear(in_features=768, out_features=768, bias=True)\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " (output): BertSelfOutput(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " (intermediate): BertIntermediate(\n", - " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", - " )\n", - " (output): BertOutput(\n", - " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", - " (LayerNorm): BertLayerNorm()\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " )\n", - " )\n", - " )\n", - " )\n", - " (pooler): BertPooler(\n", - " (dense): Linear(in_features=768, out_features=768, bias=True)\n", - " (activation): Tanh()\n", - " )\n", - " )\n", - " (dropout): Dropout(p=0.1, inplace=False)\n", - " (classifier): Linear(in_features=768, out_features=2, bias=True)\n", - ")" - ] - }, - "execution_count": 16, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" } ], "source": [ "model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=2)\n", - "model.cuda()" + "\n", + "if torch.cuda.is_available():\n", + " model.cuda()\n", + "else :\n", + " model" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "id": "HBC2Q_Q-gqDI" + }, "source": [ "## Fine-Tuning BERT" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 691 + "height": 681 }, - "colab_type": "code", "id": "1kkAaGSjqElS", - "outputId": "f927d079-5710-4348-d2d1-17c8b6194c3f" + "outputId": "29dc4911-3fba-483d-ca51-975ada380b53" }, "outputs": [ { @@ -984,7 +820,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Train loss: 0.11491870601395411\n" + "Train loss: 0.1373159165760236\n" ] }, { @@ -992,15 +828,15 @@ "output_type": "stream", "text": [ "\r", - "Epoch: 25%|██▌ | 1/4 [01:55<05:45, 115.33s/it]" + "Epoch: 25%|██▌ | 1/4 [01:12<03:37, 72.39s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Validation Accuracy: 0.9926215277777778\n", - "Train loss: 0.02726215525929417\n" + "Validation Accuracy: 0.9887152777777778\n", + "Train loss: 0.025169503974861333\n" ] }, { @@ -1008,7 +844,7 @@ "output_type": "stream", "text": [ "\r", - "Epoch: 50%|█████ | 2/4 [03:59<03:56, 118.08s/it]" + "Epoch: 50%|█████ | 2/4 [02:24<02:24, 72.34s/it]" ] }, { @@ -1016,7 +852,7 @@ "output_type": "stream", "text": [ "Validation Accuracy: 0.9921875\n", - "Train loss: 0.01315672774799168\n" + "Train loss: 0.012851784140886073\n" ] }, { @@ -1024,30 +860,29 @@ "output_type": "stream", "text": [ "\r", - "Epoch: 75%|███████▌ | 3/4 [06:04<02:00, 120.10s/it]" + "Epoch: 75%|███████▌ | 3/4 [03:36<01:12, 72.29s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Validation Accuracy: 0.9921875\n", - "Train loss: 0.009972668485715985\n" + "Validation Accuracy: 0.994140625\n", + "Train loss: 0.014206296099083764\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ - "\r", - "Epoch: 100%|██████████| 4/4 [08:09<00:00, 121.51s/it]" + "Epoch: 100%|██████████| 4/4 [04:48<00:00, 72.22s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Validation Accuracy: 0.9921875\n" + "Validation Accuracy: 0.994140625\n" ] }, { @@ -1059,12 +894,13 @@ }, { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3gAAAHwCAYAAAD0Es3SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzde7A0913f+c+3e+ac8zyPJMtGj7lY\nNjLgBJxAAdEaEm82JgVZO9mVuaSCveUsZDcYqtaBDRQVOyQuyhAwl5jaJYbgTcAEMMLYYGQQGBNj\niBWwLWOBkWQZWZKtiyXr/lzPOTPdv/2j+9e36bl39/TMeb+qpHObM6fPPHNm5tvfmznnBAAAAADY\nfsGmDwAAAAAA0AwCPAAAAADYEQR4AAAAALAjCPAAAAAAYEcQ4AEAAADAjiDAAwAAAIAdQYAHADgR\nzCw0swtm9rwmL7vCcfywmb216esFAECSBps+AAAA6pjZhcKHpyUdSYrSj7/TOfcry1yfcy6SdEXT\nlwUAoE8I8AAAveScywIsM7tP0j93zv3BtMub2cA5N+7i2AAA6CtKNAEAWyktdfw1M/tVMzsv6VVm\n9rfN7E/N7Ckz+4yZ/b9mNkwvPzAzZ2bXpR//cvr13zWz82b2J2b2/GUvm379ZWb2CTN72sx+2sxu\nMbNvX/D3+CYzuz095veZ2V8vfO1fm9lDZnbOzD5uZi9JP/+1ZvZn6ecfMbOfaOAmBQDsAAI8AMA2\n+yZJb5P0DEm/Jmks6XskXSPpxZJeKuk7Z3z//ybp30p6lqRPS/qhZS9rZs+W9HZJ35/+3HslvWiR\ngzezL5P0S5L+haSzkv5A0k1mNjSzv5Ee+1c7566S9LL050rST0v6ifTzXyLpHYv8PADA7iPAAwBs\nsw84597tnIudc5edcx92zn3QOTd2zt0j6S2S/t6M73+Hc+5W59xI0q9I+soVLvu/SLrNOfdb6dd+\nStJjCx7/KyTd5Jx7X/q9b1QSrH6NkmD1QNLfSMtP701/J0kaSXqBmX2Oc+68c+6DC/48AMCOI8AD\nAGyz+4sfmNmXmtnvmNnDZnZO0huUZNWmebjw/iXNHqwy7bJfUDwO55yT9MACx+6/91OF743T732O\nc+4uSd+n5Hf4bFqK+nnpRf+ZpBdKusvMPmRm/3DBnwcA2HEEeACAbeYqH/+cpL+U9CVp+eLrJVnL\nx/AZSdf6D8zMJD1nwe99SNIXFr43SK/rQUlyzv2yc+7Fkp4vKZT0o+nn73LOvULSsyX9e0nvNLOD\n9X8VAMC2I8ADAOySKyU9Leli2t82q/+uKb8t6avN7H81s4GSHsCzC37v2yXdYGYvSYfBfL+k85I+\naGZfZmZfZ2b7ki6n/8WSZGb/1MyuSTN+TysJdONmfy0AwDYiwAMA7JLvk/RtSoKkn1MyeKVVzrlH\nJH2rpDdJelzSF0v6qJK9ffO+93Ylx/uzkh5VMhTmhrQfb1/Sjyvp53tY0jMl/UD6rf9Q0p3p9NCf\nlPStzrnjBn8tAMCWsqRVAAAANMHMQiWll//YOfffNn08AICThQweAABrMrOXmtnVaTnlv1Uy5fJD\nGz4sAMAJRIAHAMD6/kdJ9ygps/yfJX2Tc25uiSYAAE2jRBMAAAAAdgQZPAAAAADYEQR4AAAAALAj\nBps+gGVdc8017rrrrtv0YQAAAADARnzkIx95zDlXu3N16wK86667TrfeeuumDwMAAAAANsLMPjXt\na5RoAgAAAMCOIMADAAAAgB3RaoCXLn69y8zuNrPX1nz9p8zstvS/T5jZU20eDwAAAADsstZ68Mws\nlPRmSd8g6QFJHzazm5xzd/jLOOf+ZeHy/0LSV7V1PAAAAACw69rM4L1I0t3OuXucc8eSbpT08hmX\nf6WkX23xeAAAAABgp7UZ4D1H0v2Fjx9IPzfBzL5Q0vMlvW/K119tZrea2a2PPvpo4wcKAAAAALug\nL0NWXiHpHc65qO6Lzrm3OOeud85df/Zs7boHAAAAADjx2gzwHpT03MLH16afq/MKUZ4JAAAAAGtp\nM8D7sKQXmNnzzWxPSRB3U/VCZvalkp4p6U9aPBYAAAAA2HmtBXjOubGk10h6j6Q7Jb3dOXe7mb3B\nzG4oXPQVkm50zrm2jgUAAAAAToLW1iRIknPuZkk3Vz73+srHP9jmMQAAAADASdGXISsAAAAAgDUR\n4AEAAADAjiDAAwAAAIAdQYAHAAAAADuCAA8AAAAAdgQBHgAAAADsCAK8Hvmmn7lFb73l3k0fBgAA\nAIAtRYDXI3c/ckH3PX5p04cBAAAAYEsR4PWIk+Sc2/RhAAAAANhSBHg94pwT4R0AAACAVRHg9UiS\nwdv0UQAAAADYVgR4PeKcRA4PAAAAwKoI8HrEyZHBAwAAALAyArwW/OqHPq1v/plblv6+JIMHAAAA\nAKshwGvBXQ+f1+0PnVv6++jBAwAAALAOArwWjKJYUbxCpOay/wEAAADA0gjwWjCKYo1jt/ROO3rw\nAAAAAKyDAK8F4yiJ0pZN4jlHiSYAAACA1RHgteA4iiVJ4zhe6vucWJMAAAAAYHUEeC0YpQHekvGd\nYkeJJgAAAIDVEeC1YJSWaC6dwWNNAgAAAIA1EOC1wGfwVpmkSQYPAAAAwKoI8FowynrwFo/W/MRN\nevAAAAAArIoArwW+RHOZDF6WuSO+AwAAALAiArwWjFfJ4FXeAgAAAMCyCPBacOwzeNEKJZo04QEA\nAABYEQFeC0Yr7MEjgwcAAABgXQR4LVhliqZP3JHAAwAAALAqArwWjLM9eMv04PkpmgAAAACwGgK8\nFhyvlcEjxAMAAACwGgK8FqyyB88jvAMAAACwKgK8FozGPoO3xJAVpqwAAAAAWBMBXgtGaeZuHDk9\n+NRlPX7haO735D14RHgAAAAAVkOA14D/fvdjetdHH5SU9NAVp2i+5m1/pn93851zr4MpmgAAAADW\nRYDXgHf82QP6yd+/S1IS1PkgbRw7PX1ppPOH47nXkVVoEuABAAAAWBEBXgMGgWUTM4uDVaLY6TiK\nFS8wbCV2lGgCAAAAWA8BXgPCINAo3X3nVyRISbA3jpyiBdJylGgCAAAAWBcBXgOSDF4S2PkJmlIy\nRXMcx1poW4IrvQEAAACApRHgNWAQWlaa6TN5UpLBG0VuoRLNbIomER4AAACAFRHgNaDYgzeKihk8\np3EUZ1+bJQ/siPAAAAAArIYArwFhEBQyeIUevMhpFC/Yg+ffEt8BAAAAWBEBXgPKGbzyFM3xglM0\nXTZFEwAAAABWQ4DXgDAN8IpLzqVkombs8hUIs+QZPEI8AAAAAKshwGvAIDBJScauGOAdjqLk8wvE\nbI4pmgAAAADWRIDXgDBMAjw/NdM7SlcmMEUTAAAAQBcI8BrgM3jjaRm8RRbhkcEDAAAAsCYCvAaE\nQXIzRlF9gEcPHgAAAIAuEOA1YJiVaMalEs3DUVqiuUiAR1wHAAAAYE0EeA0I5w1ZoQcPAAAAQAda\nDfDM7KVmdpeZ3W1mr51ymX9iZneY2e1m9rY2j6ctU3vw/JCVpaZoEuEBAAAAWM2grSs2s1DSmyV9\ng6QHJH3YzG5yzt1RuMwLJL1O0oudc0+a2bPbOp42ZT14lSmay2Xw0rfEdwAAAABW1GYG70WS7nbO\n3eOcO5Z0o6SXVy7zHZLe7Jx7UpKcc59t8Xha08QUTT9cZZF+PQAAAACo02aA9xxJ9xc+fiD9XNFf\nk/TXzOwWM/tTM3tpi8fTmrwHLy4FeEfpkJVFJmNmJZrEdwAAAABW1FqJ5hI//wWSXiLpWkl/bGZf\n7px7qnghM3u1pFdL0vOe97yuj3GuYgbveFzswUszeMsEeM0fHgAAAIATos0M3oOSnlv4+Nr0c0UP\nSLrJOTdyzt0r6RNKAr4S59xbnHPXO+euP3v2bGsHvCqfwRtHTuO0HPPUMCyUaM6/DsemcwAAAABr\najPA+7CkF5jZ881sT9IrJN1Uucy7lGTvZGbXKCnZvKfFY2rFICz04KUZvFN74Up78JiiCQAAAGBV\nrQV4zrmxpNdIeo+kOyW93Tl3u5m9wcxuSC/2HkmPm9kdkv5Q0vc75x5v65jaMsimaOY9eHthwBRN\nAAAAAJ1qtQfPOXezpJsrn3t94X0n6XvT/7bWoFCiOYqd9sJAg9B07vJYkhQvMUWT+A4AAADAqlpd\ndH5S5FM0kxLNYWgaBLZciaZ/SwoPAAAAwIoI8BpQ6sGLYg0HgcLAdJyWazJFEwAAAEAXCPAaEGY9\neE7HkdMgCLK+PEmKF5ii6UM7EngAAAAAVkWA14DiHrxxFGsvtKxsUyKDBwAAAKAbBHgNyHvw4qxE\n05dtJp9fvAePFB4AAACAVRHgNcBn8EaR0yhyGgTlDJ40f3gKGTwAAAAA6yLAa0BxiuZxFGsYBlnQ\n583L4jl68AAAAACsiQCvAcMwuRmzHrx0imbRvD68PINHhAcAAABgNQR4DSj34Lk0g1e+aedN0swC\nPOI7AAAAACsiwGtAcYrmcRTX9uDNzeBRogkAAABgTQR4DSj24PkSzWoPXsyQFQAAAAAtI8BrgC/H\nHKdTNIfhZA9ePG/ISlaiSYgHAAAAYDUEeA0IwzyDN5pWorngFE0AAAAAWNVg0wewC6o9eMNBIKtc\nZuEpmsR5AAAAAFZEgNcAn60bR7HGkdNeGEyUWs6dopm9JcIDAAAAsBoCvAaElmfwRlGsYWiK4mWH\nrDBFEwAAAMB6CPAaEASmwAo9eGEgq2Ti5vfgld8CAAAAwLIYstKQQRhoHDsdjWLtD4Js8Iq38JoE\nUngAAAAAVkQGryGDwBTFsY6iWPuDcCJjNy+D53N3hHcAAAAAVkUGryFhYBpFTsfjNIO34qJzIjwA\nAAAAqyLAa8ggMF0+jiRJ+8MgW53gRQtP0QQAAACA1RDgNSQMAl08HkuS9gehwqB809KDBwAAAKBt\nBHgNGQSmS2kGb29Ql8FbcE1CO4cHAAAA4AQgwGtIGJguHvkM3go9eP4tER4AAACAFRHgNWQQ5hm8\n/UIGz8d58zN46VtyeAAAAABWRIDXkDCwcg9eugdvfxBKWiSDl5ZoEt8BAAAAWBEBXkOGQaBLR5NT\nNPeHyU284Bo8AjwAAAAAKyPAa0g5gxdkUzQP0gzevBLNuQEgAAAAAMxBgNeQaT14e4M0gzevBy8r\n0STSAwAAALAaAryGhIFlWbpkD57vwUtu4mjRPXjtHSIAAACAHUeA15Di3rtiBs/34M2dounfEuEB\nAAAAWBEBXkPCUoBXzOAlPXjzArd80TkRHgAAAIDVEOA1ZBDkN+X+MNAgrJRoksEDAAAA0DICvIaE\nlRJNP0Vz0R480YMHAAAAYE0EeA0p9uDtrTVFs6UDBAAAALDzCPAa4ksyJWkvDCZ68BadokkODwAA\nAMCqCPAa4nvwBoFpEBamaPoM3twhK+W3AAAAALAsAryGVPfehZU1CfNLNMtvAQAAAGBZBHgNyffe\nhenHfshKWqI5L8DzaxJI4QEAAABYEQFeQ6Zl8PYWnKJJBg8AAADAugjwGuKHrOxVSzQXnaJJDx4A\nAACANRHgNWRqD15aojlvyIpEiSYAAACA9RDgNaTac1edornomgTCOwAAAACrIsBrSDWD50s1T++l\nGbw5KbyYJjwAAAAAayLAa4jvwfNrEb70867UG7/5y/V1X/psSQtM0fQlmi0eIwAAAIDdRoDXkEGl\n587M9IoXPU8HQ9+Dt+iQFUI8AAAAAKshwGtImPXgBZXPJ4Hf/Axe+S0AAAAALIsAryGDyt47L7Tk\n8/OmaOaLzps/NgAAAAAnAwFeQ6pDVrw0sTe3RNNzM3J4N37o03rnRx5Y7QABAAAA7LxWAzwze6mZ\n3WVmd5vZa2u+/u1m9qiZ3Zb+98/bPJ42VXvwPJ/Bm1uiucCi87ffer/e+WcEeAAAAADqDdq6YjML\nJb1Z0jdIekDSh83sJufcHZWL/ppz7jVtHUdXpmXwFu/Bmz9F04kSTgAAAADTtZnBe5Gku51z9zjn\njiXdKOnlLf68jcoyeMPyTWpZD95iGbxZEV7sFi/1BAAAAHDytBngPUfS/YWPH0g/V/UtZvYXZvYO\nM3tui8fTqjD0UzTDya8FtviahBkRnnOzvgoAAADgpNv0kJV3S7rOOfcVkt4r6RfrLmRmrzazW83s\n1kcffbTTA1zUcMoUTSnpw4vi2d+fJfBmRHDOsScPAAAAwHRtBngPSipm5K5NP5dxzj3unDtKP/xP\nkv5W3RU5597inLveOXf92bNnWznYdU3rwZOSSZrzM3jze/Bi5+jBAwAAADBVmwHehyW9wMyeb2Z7\nkl4h6abiBczs8wsf3iDpzhaPp1WDsH6KpuQzeAsuOp8RwdGDBwAAAGCW1qZoOufGZvYaSe+RFEr6\neefc7Wb2Bkm3OudukvTdZnaDpLGkJyR9e1vH07Yw8D14NRm8BQI8udKb+ovQgwcAAABghtYCPEly\nzt0s6ebK515feP91kl7X5jF0ZdoUTUkKApvbO5etSZjTgzcvTgQAAABwcm16yMrOCKcsOvdfixZd\nk6DpZZpxMmVl9YMEAAAAsNMI8BoymDFFM1hiiqY0PYaLnSODBwAAAGAqAryGzJqiGQZSPCcyKw5P\nmXZJ52bvyQMAAABwshHgNeQrn3u1vvmrnqO/+ZxnTHwtsGZKNJ2keE4mEAAAAMDJ1eqQlZPk6tN7\netO3fmXt1wKz+XvwprxfFDNFEwAAAMAMZPA6EAY2t0SzmMKb1YM3bxonAAAAgJOLAK8DyRTN2Zcp\nZ/CmTNGMGaIJAAAAYDoCvA4ENn/ISrkHb/rl5pV6AgAAADi5CPA6EAamaG6ANz9wowcPAAAAwCwE\neB1YesjKzD14hHgAAAAA6hHgdWChAK9YojmtB8/RgwcAAABgOgK8DixUoll8f8pFnVuslBMAAADA\nyUSA14FgkSmaxTUJMy4zb9sCAAAAgJOLAK8D4QJTNIumZemSIStEeAAAAADqEeB1ILBFpmgW3p92\nGSW78AAAAACgDgFeB4JgkSmahRLNaVM0qc8EAAAAMAMBXgfCJadoTkvhOceicwAAAADTEeB1YOkp\nmlPXJDjWJAAAAACYigCvA4tM0Sxm5qauSRAZPAAAAADTEeB1IFhgiuYiQ1aSKZoAAAAAUI8ArwOL\n9OAVTV+TwKJzAAAAANMR4HUgWKQHb8FF58R3AAAAAKYhwOvAslM0p/bgMUUTAAAAwAwEeB1odIpm\ng8cFAAAAYLcQ4HUgWXQ++zKL7MGLHcvOAQAAAExHgNeBwOaXVhZzc3WX9D16hHcAAAAApiHA60Bo\niwxZqX+/+jla8AAAAABMQ4DXgSCw+XvwSu9PXtZnAFmTAAAAAGAaArwOhGaK5gVmxTUJNRf18SEt\neAAAAACmIcDrQLIHb/Zl5s1Y8Rk81iQAAAAAmIYArwOBzS+tLPfgTb8s4R0AAACAaQjwOhAG80s0\nS1M0a0s06cEDAAAAMBsBXgeCJado1omZogkAAABgDgK8DoTLTtGsXZNADx4AAACA2QjwOrBQiWax\nB692TYL/GgAAAADUI8DrQGCmeN4UzTlrElzWg9fkkQEAAADYJQR4HQhsfmnl/DUJha8T5QEAAACo\nQYDXgcVKNF3t+3WfY9k5AAAAgDoEeB0IzOTcnP12pR68SWTwAAAAAMxDgNeBMDBJmrkqYdEpmhIZ\nPAAAAAD1CPA6kAV4C2bw6nJ48ZwpmwAAAABAgNeBwBbJ4M2Zojnn6wAAAABAgNeBa67YkyQ9ev5o\n6mWW68Fr6MAAAAAA7BQCvA5c+8zTkqQHnry80OXrArg4LvbgEeEBAAAAmESA14Frn3lKkvTAk5em\nXqa0JqEmhzcvwwcAAAAABHgd+LxnHCiw2Rm8uVM0RQYPAAAAwGwEeB0YhoE+/xmnZgd4c3rs6MED\nAAAAMA8BXkee88xTs0s0NbtEs5i1Y9E5AAAAgDoEeB259pnrZfBYdA4AAABgnlYDPDN7qZndZWZ3\nm9lrZ1zuW8zMmdn1bR7PJl37zNN6+Nyhjsdx7dfnxWzlAJAIDwAAAMCk1gI8MwslvVnSyyS9UNIr\nzeyFNZe7UtL3SPpgW8fSB9c+85Sckz7zdH0Wb5kePDJ4AAAAAOq0mcF7kaS7nXP3OOeOJd0o6eU1\nl/shST8m6bDFY9m4fFXCtDLNJXrwWJQAAAAAoEabAd5zJN1f+PiB9HMZM/tqSc91zv1Oi8fRC9de\nnSw7f/Cp+gAvLlRu1vfgzf46AAAAAGxsyIqZBZLeJOn7Frjsq83sVjO79dFHH23/4Fpwai+UJB2N\notqvl6doTipP0Wz00AAAAADsiDYDvAclPbfw8bXp57wrJf1NSe83s/skfa2km+oGrTjn3uKcu945\nd/3Zs2dbPOT2DEOTJB1H9dHZvCEqrtSDR4QHAAAAYFKbAd6HJb3AzJ5vZnuSXiHpJv9F59zTzrlr\nnHPXOeeuk/Snkm5wzt3a4jFtzDBMbupxNH+K5twMXoPHBQAAAGB3tBbgOefGkl4j6T2S7pT0dufc\n7Wb2BjO7oa2f21eDNIM3njICc+4evML7MWM0AQAAANQYtHnlzrmbJd1c+dzrp1z2JW0ey6YNgySW\nHk3N4M3O4VGWCQAAAGCejQ1ZOWmCwBSYNJ7Sg6d5GbzCJwn2AAAAANQhwOvQMAxmZPDq3/di1iQA\nAAAAmIMAr0NJgDetB2/2GgSmaAIAAACYp9UePJQNQtM4XiCDVwjgnrx4rI8/fF5m9ZcFAAAAAI8M\nXocGwawMXv5+sRzzVz/8af3vP//BUu9e3Z48AAAAACDA69AwtAX34OUfHR5HGkWulPkjvgMAAABQ\nhwCvQzOHrJQW4eXvRunnixk81uABAAAAqEOA16FBaBpNW3Q+5X0fDxYXpDu68AAAAADUIMDr0DAI\nppZoTtuDF6WlmcUSzSlzWgAAAACccAR4HRqENnXReTErV3zfx4NRXCzRJIMHAAAAYBIBXocGYTC1\nRLOYlavbeTdt+iYAAAAAeAR4HdqbOUWzmMHL+dLMqFiiSQYPAAAAQA0CvA4le/CmTdEsvj9Zolka\nskJ8BwAAAKAGAV6HBqFNX3Q+5f04rluTQIQHAAAAYBIBXoeGYVCahlnkpkR4PnNXXpMAAAAAAJMI\n8Do0CKZP0dSUKZpxtug8DwwdGTwAAAAANQjwOjQcLNqDl78f1WXwiO8AAAAA1CDA69AwWLAHrybA\nK+/Ba+PoAAAAAGw7ArwODcJg+poE52SWvl/4fJbBo0QTAAAAwBwEeB0ahjZ10bmTFKQRXmlNgl90\nTgYPAAAAwBwEeB0aBLMyeFJQk8GLa0o0HXM0AQAAANQgwOvQMAymTtF0kizL4OWfH9fswaNCEwAA\nAECdhQI8M/tiM9tP33+JmX23mV3d7qHtnmFoOp7Rg+czeKpbkxAXe/DaOkIAAAAA22zRDN47JUVm\n9iWS3iLpuZLe1tpR7ahBaKV1B1VBTQbPl2YWp2/GRHgAAAAAaiwa4MXOubGkb5L0086575f0+e0d\n1m4aBIGi2NVOwUx68NIAr/D5cdaDV8jgtXqUAAAAALbVogHeyMxeKenbJP12+rlhO4e0u4ZhEsDV\n7cJzKqxJKHw5rll0TgYPAAAAQJ1FA7x/JulvS/p3zrl7zez5kn6pvcPaTcMwubmL/XReOYM3uSah\nNJyF+A4AAABAjcEiF3LO3SHpuyXJzJ4p6Urn3I+1eWC7aJAGeHUZvLgwZKWuBy8igwcAAABgjkWn\naL7fzK4ys2dJ+jNJ/5+ZvandQ9s9eYlmfQYvDCZ78KJ4coomi84BAAAA1Fm0RPMZzrlzkr5Z0n9x\nzn2NpK9v77B20yBISzRre/CKe/AKJZq1e/CI8AAAAABMWjTAG5jZ50v6J8qHrGBJgxkZPDkV9uDl\notohK60cHgAAAIAtt2iA9wZJ75H0Sefch83siyT9VXuHtZv2siEr9VM0a/fguckePKasAAAAAKiz\n6JCVX5f064WP75H0LW0d1K7yGbzxlB68uimacbbonB48AAAAALMtOmTlWjP7TTP7bPrfO83s2rYP\nbtf4HrzjugBPqt2Dl61JiIs9eK0dIgAAAIAttmiJ5i9IuknSF6T/vTv9HJYwzDJ4NSWabkqJZsSi\ncwAAAACLWTTAO+uc+wXn3Dj9762SzrZ4XDtpMGvRufIhK6U1CVkPXly6LAAAAABULRrgPW5mrzKz\nMP3vVZIeb/PAdlG+B68ug1fowSutSUjesiYBAAAAwDyLBnj/h5IVCQ9L+oykfyzp21s6pp01DOft\nwcvf93zmjh48AAAAAPMsFOA55z7lnLvBOXfWOfds59w3iimaSxsEs/bg5T14xQgvX3RenKJJhAcA\nAABg0qIZvDrf29hRnBA+g1cX4CU9eDVrEtJ3yeABAAAAmGedAM8aO4oTItuDV7fo3NWvSfADWSKm\naAIAAACYY50AjyhjSbMzeIU1CYXP++GZxcEs3PAAAAAA6gxmfdHMzqs+njBJp1o5oh02DGYMWXFS\n+uUpi84LaxLI4AEAAACoMTPAc85d2dWBnAR5ieZkBi920jAo9+A557LSzCiiBw8AAADAbOuUaGJJ\nPsA7rs3gFUo00y/HpV68ycErAAAAAFBEgNehvWwPXs2aBElBZUtCcbBKqUSTLjwAAAAANQjwOjSY\ntejcqbAHL/l6cVpmccgKGTwAAAAAdQjwOpQtOq/pwaubolm3TkFiyAoAAACAegR4HRrOyeBV9+BF\nUwO8Vg4PAAAAwJYjwOtQGJjMpu3BU2HISlqiOSXAY9E5AAAAgDqtBnhm9lIzu8vM7jaz19Z8/bvM\n7GNmdpuZfcDMXtjm8fTBMAxK/XSecy7fg5d+bnqJZksHBwAAAGCrtRbgmVko6c2SXibphZJeWRPA\nvc059+XOua+U9OOS3tTW8fTFMLDaKZrlDF7yuWqmzk/ZJIMHAAAAoE6bGbwXSbrbOXePc+5Y0o2S\nXl68gHPuXOHDM9Luz/8fhEF9Zs5JVhmyUu3BC32EBwAAAAA1Bi1e93Mk3V/4+AFJX1O9kJn9X5K+\nV9KepL9fd0Vm9mpJr5ak5z3veY0faJeGoc3owUvfTzN01QAvyfA5MngAAAAAam18yIpz7s3OuS+W\n9K8k/Zspl3mLc+5659z1ZxKrRJcAACAASURBVM+e7fYAGzYIgvoAz+VrErxpGTziOwAAAAB12gzw\nHpT03MLH16afm+ZGSd/Y4vH0wnBg9WsSVMzgJW8jVx/g7fqi88+eO9Q/+Kk/0oNPXd70oQAAAABb\npc0A78OSXmBmzzezPUmvkHRT8QJm9oLCh/9I0l+1eDy9MAwCjWoiNFfqwasv0cwyeDveqnjvYxf1\niUcu6JOfvbDpQwEAAAC2Sms9eM65sZm9RtJ7JIWSft45d7uZvUHSrc65myS9xsy+XtJI0pOSvq2t\n4+mLQThtiqabzOBVAzw7GSWa8ZTfHwAAAMBsbQ5ZkXPuZkk3Vz73+sL739Pmz++jpAevPoMXzJmi\nGQTlRei7atqQGQAAAACzbXzIykkzDE3juG7ISjGASz5XnZY5OCE9eP73m7boHQAAAEA9AryODcPp\nUzTDSg9eNcCpLkLfVT6wZR0EAAAAsBwCvI4NQtNovNgUzXjKkJVdD3z870cGDwAAAFgOAV7HTu8N\ndHkUTXy+2IPnTZ+iudumBbgAAAAAZiPA69jpvVAXj8cTn3dy+ZqEKUNGzJL/dn3Iiv+9yeABAAAA\nyyHA69gV+wNdPKoJ8Nz8ReemJMu34/Fd3oNHgAcAAAAshQCvY6f3Brp4VFOiqQXWJJjJdBJ68JK3\nZPAAAACA5RDgdeyK/aREs1pmmaxJyN+X6gO8wGzn1yTkJaqT00YBAAAATEeA17HT+wM5p5pBK4Ue\nPE3vwZPlX99V8ZQAFwAAAMBsBHgdO7M/kKSJMk3nkh47KQ9wqqWYZqbATs4ePEo02/HIuUP6GwEA\nAHYUAV7HzuyFkjQxaMUpzdBJWQTnA5y9MPlnCkwy2c5P0WTReXuevHisv/tjf6g/uPORTR8KAAAA\nWkCA1zGfwbtQDfCck8mSNQjp53yJ4jBMIr8gzeDtevLFMWSlNReOxjqOYj124XjThwIAAIAWEOB1\n7MxeEuBdOq6UaCrdc6fCou/0neGgkMFjTQLWkJe/MsAGAABgFxHgdezM/pQSzbQHz8wKQ1aSrw3T\nEk1ZkuHb9dJF1iS0J1siH3HbAgAA7CICvI5d4YesHNeUaKZ77vI1CUmEV+7B231k8NrDhFIAAIDd\nRoDXsdPZFM36ISvlHrzkbakHL7Cdz+A5pmi2xt+2I0o0AQAAdhIBXseu2PNDVip78FwyITOZkpl8\nKvI9eJUM3o7Hd3mWadd/0Q3wt2lEiSYAAMBOIsDr2Om0B+9SJYMXOzexyDxKU3g+wDOZAtv9DJ4v\nHyQIkW697wm96b2faOz6fOJuRHYUAABgJxHgdWwYBtobBLpQ7cFTOmTFfyDJxzd+ima1hHNX+TJC\nMnjS79/xiP7jH32ysevzJwciSjQBAABmet1vfEzfc+NHN30YSxts+gBOojN7oS5VSjSdmwzg4mzR\ned6Dl6xJ2O3Ah0EguTh2jf57Z2sSyI4CAADMdP8Tl3SpkpTZBmTwNuDM/qBmyIqfomkTQ0ayEs0T\n04Pns0w7/osuIHKu0cX2rKAAAABYTOycwmD7ZtgT4G3Amb2BLkzdg1ez6DwbsrJ4D96DT13W05dH\njR53V8jg5Zxr9nbI9+BRogkAADBLnK4x2zYEeBtwZj/UpeNKiaYk+Qxd+rmoJoMX2GIZvBe/8X16\n8Rvf19gxd8mRwcv426CpMk1WUAAAACwmjpPX3tuGAG8DzuxPZvCyNQmWr0nISzTLPXiLvja/cDTe\nymXhlGjmsqXvDd0UZEcBoL/uf+KSfuTmO7fyuRvYRZRoYmFn9gYTDZtJD57P4KUv6mOnwKQgvWPl\nQ1gWf+C/8+FzjR13V9iDl2s62PXXM2LICgD0zvs/8aje8sf36NELR5s+FABKXocFlGhiEcmQlZop\nmlKyB68Q4AyCQKEVM3jLDVm55e7Hmjno1INPXW49+xNTRpjx2wya2n2Yl7/SgwcAfeOyqg2e/4A+\niJzowcNizuyHk0NWlE/J9OLYKQjy2t/AtPCQlatPDyVJH7j78YaOWnrq0rG+7ifer9/+i4cau05J\nunQ81n/6b/dkJSlZgEuWKfu3buq53sfMLDoHgP7xJ1Apowf6wTmncPviOwK8TTizX1Oi6VyhBy/P\nYIVmhRJNW3hNgg+Wbr3vicaO+/GLxzqOYt3/xKXGrlOS/vgTj+qHf+dO3fXIeUn5sVOimd8GTd0W\n2fURPANA7/i4jqc/oB8o0cTCzuyFGkVOR+O8TDPL4Fl5imYYWFaiaVo8g+efJC4dR41NYPS7+564\n2Oz6hXE2ur88UIQzmJMrM9aVl79SogkAfUOJJtAvUUyJJhZ0Zn8gSbpU6MPL9uCp/KI+DCyb3hOk\nNZx1D/s//4F7dftDT2cfF4OjpgZq+LLSJy8dN3J9XlTJ2DFFM5etSWgoHmNNAgD0FyWaQL845xRu\nYbS0hYe8/XyAN7EqIV2D4KdkjtMMnj9zkPTjWW1G7kduvlO/dVveG1c8+9fUE4UPSJ+42GyAl/Xc\npVkl9uDl4qZLNGP/ltsWwPo+8ch53ffYxU0fxs6IXfktgM2iRBMLu+ogCfDOHSaljj6gCaySwfMl\nmum/0rQevHEUaxy70ov22DntDZJvHDVUjnfxuOUMng8+CPAyccPlOv56RhElmgDW99p3/oXe+Lsf\n3/Rh7IymH/MBrCeKCfCwoCsPkgmX5w+TgMk/jidDVio9eDa/B+9oHGeXV+F799MAb9xwiWbTGbxq\nXxg9eLn21iRw2wJY3+VRrMujaP4FsRA/ZIwAD+gH5/J91NuEAG8DrqoGeOnnLV2E5woBTlAs0Zyy\nB+8wfXItZ/BUCPAayuD5HryWAryo8sTGFM3C2dyGEm7+rsCicwBNcM4RjDSIE5xAvyQlmps+iuUR\n4G3Alb5E83JSoumfHE0+yMsDnPKQlaRMs/q4n2XwsmAgebs/CCU1t/PsQtqDd/E4yoLKJlT7wvxr\nBQaBtFeiyYsHAE2IYgK8JvnncW5SoB8ievCwqKtO+Qye78FLPm+VHrxsTUJlD151jqYPtqr745rO\n4F0qDIVpsg9vIoNXeXuS5Q339OAB6J/YucYqDMCaBKBv4lgEeFhMlsHLSjTTDF6lBNP34Pn7lVky\nSXNqBq9S4pgNWWmoHO9iYTl7k314kyWayefJ4BWD3maujwwegCbFjnL6JrEmAegXR4kmFjUMA50a\nhhMZPCkZtOIDvuqi88Csdk1C1oNX6dfKMngNRQcXCnv7nmxw2XkcVwM8Mnhe4yWa6V2B4BlAE2Ln\nalf3YDWsSQD6JXZk8LCEKw8GOn84VlRYb2CmUgZvctF5UsJZfeA/HJUzeFElg9fUFM2LR2MdDJPr\nfKLBEs2okrHLl3FT99N0gBdx2wJoUPE5DOtjTQLq/PKffop9kx167x2P6MVvfJ+Ox3HSg7eFKTwC\nvA256tRQ5w5H+uHfuUOv+s8flJSuSVBlTUJhimayRsFUfdg/GpenaE6WaDaVwRvruc88LanZSZrV\nngPOYOZaW5PAFE0ADYhjJx5OmkMPOqqi2OnfvOsv9a7bHtz0oZwY9z52QQ8+dVmXjyNKNLEcn8G7\n46FzeuDJy5J8Bs9KUySDwh68IPAZvmqJpt8fVz9Fs6lyvEvHY33B1adk1mwPng9MfaaxuhfvJKsG\nvetfX/K2qcmqAE622E0+J2F1nOBEFX2Z3cumuztHiSaWc9XBUOcuj/TIuUMdp0NS/N3H9+DFzmkQ\nmML0X8myHrzydVUzeFEW4DWbwbt4FOmqU0M949Sw0SmavmywOmSF+K6FEk2eKAA0KHKUaDaJEk1U\ncZ/oXvE299V024YAb0N8Bu+Rc0dZAOZ78HwNpl90Hkz04JX/yI+yHrzkY/9c23QP3oWjsa7YD/Ws\n03t6vNESzeRtvv+HDJ7X9NJbf9uyJgFAE1x6hhvN4MU8qvLp1xs+kBOkWCodO6ctTOBpsOkDOKmu\nOjXUQ09f1uEo1iAo9tiVe/AGQZClhk31GbzDNINXfWJoeormxaOxzuwN9Mwze3riQgslmpUeQh7M\nJpe/r6vpgBHAyRbFjn6xBlFlgarqfAW0L6ssc06OEk0s48qDQdY75wObZNF5vgZhck2CJJuewRtP\nlGgmPXhN7MGLY6dLx5HO7A/0OWf2WtmDl69LUPqWCK/ppbfVfkf0h2PcPLZQ7Hjh2SQf13GTwsvb\nVrhTdCXL4DlRoonlXHUwrP18KYPnpLBQopn04E0+8Ps9eHHlLE+TJZqX0p9xxf5AZ6/c16MXjta+\nTi+uZPDyEk0ezKr9ietigE0/jaNYX/Mj/1W/ddtDmz4UYCnJFE0eq5viGn7Mx/bLTn7zd9aZqJB4\n2NYSTQK8DbnqYLI61ixdk5CV0cVJgJfesQKz0iJ077C6JqGFRecXj8aSpNP7oa65Yl9PXDxurI8r\nLvy+yceMifbiSl/luvIdi9y+fXLxONJnzx/pvsfZc4TtEqclTGgG5XioinhN1LliOwslmljKlTUZ\nvMCvSUg/juLkThVmGbxkVUL1bzwbsuLKZ3mKJZq/fuv9+r2//MzKx3shDfB8Bk+qX5XwH//ok7r1\nvieWuu48S5V87H8/Mnj5k3xTpXvFM4Dcvv1xlGbIfdk2sC2Yotks1iSgqul1SZivmDWNnMtapbYJ\nAd6GXHWqJoOX/ucKZ2sGgWVnDoJKj56XDVmZWqIZ6xduuU9v+9D9Kx+vz+Cd2csDvEfPl8s0o9jp\nJ95zl2768+XKzPLeu0oGjzOYhYEzzZZoNnmdWJ8P7Hy5NbAt6MFrFs9/qIoLO9nQjXzAXVKiuYUt\neO0GeGb2UjO7y8zuNrPX1nz9e83sDjP7CzP7r2b2hW0eT5/UZfAsmbKSZfDGWYmmD/CSKZvzMnj5\nonNfoul0HMUar1FSefEoeeF5Zn+ga66oD/AePX+kKHZLZ4aqUzOLi95PuqbP5hafH0b04fXG5TSw\nOxrzb4LtEjNFs1EEeKiiRLN7eYlm8rrJyODlzCyU9GZJL5P0QkmvNLMXVi72UUnXO+e+QtI7JP14\nW8fTN3VDVtL4LovwYicFlUXnxRJO73Ds9+BVSjSHeQbveByvNWzlYqFE89k+g1cZtPLgU5eTn7/k\nz6lOzczLEnlAiwtnkZpQzNot+++E9vjM3REZPGyZmD14jWKVDaqq1Vlon7+t/awJpmiWvUjS3c65\ne5xzx5JulPTy4gWcc3/onLuUfvinkq5t8Xh65cq6ISvyPXh5WV6xRNNM6RTNSonmqH7Iyl5YyOCN\nYx2vk8E7Lg9ZkSYzeJ95+nL285aRT3acfBA76SUJWXazoduheNuSwesP/zfsy62BbeCXnJ/0x+km\n5Sf1Nnwg6A12A3evup95C+O7VgO850gqNn09kH5umv9T0u+2eDy9ctWpJIP3rDN7+Scnpmi60pCV\nwFT6undUyeBli86H+ZCV4yhea5pmccjKqb1QV+wPJgK8h3wGb8mfUw1iivHhST+LWXebrIPbtp98\nFp4hK9gm2VReHksaQ4kmqpis2j3/2su3NlGiuSIze5Wk6yX9xJSvv9rMbjWzWx999NFuD64lZ/ZC\nfcfffb6+8SvzmDfJ4JUDvDBQqQcvMJv4I8/24FUGcuwPVi/RvOOhc3rrLfdmH18q9OBJ0tkr9/XY\nhWqAd5j8vFV78KLJcsSTHoTkaxIayuAVbk+WnffH5WPfg0cGD9sjIhhpXNZqccKf+5DLe/G5T3TF\nZSWayVtKNMselPTcwsfXpp8rMbOvl/QDkm5wztVuz3bOvcU5d71z7vqzZ8+2crBdMzP9wD96ob7q\neVcXPqfSnrvIOYVBkC86T79v0QyeL9EcrVCi+V2//BH94Lvv0GfPJ0GbDyIP0qDx7BX7MzJ4q/Xg\n5SWaha+d8Ae04kTVJhSfIBhi0x8+sKvL4J07HOm7fukjEydUgE0jGGmevylP+FMfCpqepo358hLN\n5Dl5C+O7VgO8D0t6gZk938z2JL1C0k3FC5jZV0n6OSXB3WdbPJbeGob5P4HJShm8OM3ghVkPnp+i\nWf4jP8oyeOn3pV8PAtMgMI2iOJ2iufiDg1+F8CeffDz5GeNYg8A0SI/3miv3JoasPLRqD14lMC31\n4J3wLFPTJZpRKTtKOWAT3nrLvfrZ939yrevIevBqhqzc9fB5/d7tD+tjDz691s8AmuYIRhrnT+qd\n9JObyFGi2b0s8RD5Hrzti/BaC/Ccc2NJr5H0Hkl3Snq7c+52M3uDmd2QXuwnJF0h6dfN7DYzu2nK\n1e2sYZjfafz9x/8Jj+NkuaI/cxAU3i/yLwr9mQZ/xwzNNAgtK/9aZk3Cl33+lZKkD/zVY9nP8CWf\nUpLBe6w6ZCUt0Vz2LNPkkJX8ayf9Sa7p0ozSmoQTHjw35b13PqLfu/3hta7DZ+7q1iTkw5P490K/\nRD0NRt72wU9nFSXbhrJXVFUH6KF91Sma2xjgTY5ybJBz7mZJN1c+9/rC+1/f5s/fBuUMXrkEM46d\ngsCyEk2/6Hwig5e+KKz2awWBNAwCXUonYB4v8YLe/4hb7n5MzjkdjeNsaIuUZPjOHY51OIp0MAx1\nOIr0+MVjSctn8Ko7XujBy8UNv7gvXs9Jv22bEjWwB+zyjAxeHJdPgAB90ceBIBeOxvrXv/kxve5l\nX6rv/HtfvOnDWVrTu0+x/bK5DD36O9t11cQDJZpY2qCSwUs+ys+KDgIrlGgmQVv1b3xyTUKeUh6E\npovH5QzfIvx1PfT0oe597KKOxuUMnl+V4IO6zzx9mH1t2YXq1cXmEUFIJm6xRHO0gzOXP/X4Rb38\nP3xAT18adfYz43j9+2leolmTwaP/Aj0V9zCzcJye8NzWEyJNn9TD9mPReff8822WwdvCCI8Ab8P2\nZvTgjbMMXvp1s9oMnn9RWC2XCQNTGAS6lK44GNWUf01TfDF518PnkwxeIcC7+nSy3uGpS2mAl5bD\nhIEtn8GrBHYxGbxM07X3xavZxdv2zs+c058/8LQ+/cSl+RduSOTc2v8+eYnmZAavuo8H6Is+Tvfz\nL8i29cVwH7Oi2KzsNRL3ic5kGTx68LCqQSHAk6VrEtIP46wHL1+TUPy6pLR8MsouL+VPuoGZhoUM\n3miJJ7ziA8nlUaSjUaz9QV6iefXpZI+fz5T4PXlXHgxW7sGLKsdf/NxJ5Rp+AVVadL6DPXjjDTwR\nRrFbO/jyGbyjmgxezNlb9FQfX3j6DF6fjmkZTExElWu4kgfzTU7RJMDDkkpDVpSuSShk4gZBZdF5\nZU3CKHLZH301jR9YUgLqe/CWKZ2MYqeDYXL3OBzFSYnmsJjBSwK8py6PsstLye69lffg1fTgnfSs\nRdNnc3e9/DXPBndXM9ZED54P8I6jeOLfpbpGBOgL/1jtXPlxe5P8OqBtfXzzD109uTnRAwza6p6/\nqUdR/np62xDgbVhpyEohQxfHTs4ldb9BsQfPyk+kxZIuv1LAPxiEgSVDVo7yNQqLPulFsdMV6VLz\ny6NIR+NYB8UM3ilfopkEeD47eDAMl35xPVmimX/tpJep5EF7M9dXvG1HfWqcaUh+X+r2Z66bLSgO\nVzmulFJvImiVkseZv3jgqU5/JrZL8X7fl9eeo20P8Ho6mRSbQx9297LhZvTgYVXFAC9Ih6w4V+ij\nq5ZoqvxE6nt3DobBxHjlfMjKOLv8ooM1otjpTBrgHY6iZE1CbQbvOL18cr37g2CpfXsq/D75moRC\nBm8HywiXUc3Orqs0oXQHb9tNLF6OnWtgyEpceL/ch7epHryPfOpJ3fAfbtHHHz7X6c/F9uhjOf1o\nvN0BEmsSUNV0qwbm83+H+RRNAjwsaWIPnpmcClm4MC/RtDTYK/6R+xeDp/cGhQxYIcArZPCk5QK8\nU8NQZski9eqQlYNhqL1BkPXg+TR2ksFbMsCrZCj6+KJhU+pWR6yjeHvuYsnfJgK8KG4gwCtk4g8r\ng1Y2NUHt3GHyt33+cDznkjipivfJvrz4PI7KPenbxh92T25O9ACLzruXT9GkRBMrGlanaCp5MR8X\nMnj+IkHSpFd64Pc78E7vhYVetuRrYeCHrOQv0BbNiMXOKQxMB4MwK9EsDlmRpKtPDbMSzWIP3spD\nVrIntkKW6YQ/oDW9JqF4PcuszdgWm1i8HDWQwbt8nAd11UErm9qD5x8rTnoWHdMVX3D25cXn8bj8\nPLhtsh78LQ1Q0TxKNLuXTbNPH0jCLYzwCPA2rNyDl2bxlL+YCwOT+R48JVk5V5PBO1OTwQuDZErn\nKn1X4zgN8IZBPmRlUL67XH16qKfTISv+j+BgGK6w6Dx9m2XwXLY+ouu+oz5JAv3k/aYe2Hd9ifwm\nmtHjuIE1CYW+u4kM3gaykhLj2jFfVMrgbfBACrI1CVt6vyVbgyqmaHYvy+Clb40STSyruOhcynvw\nisvK/aLzIEgzfIXL+yErp/ZCxa6c/TMzDSpnHRYdjR/FToGZDoZhviZhWAnwTu1lPXjjQgZv+UXn\n5UxBHOe3y7aehW1C8fm9sRJN57KTCLu4JmET/WpNZPCORlF2UqO67HxTZ2/HG8ocYnv0sZx++4es\npG+39PjRPP86iKC/O1kPnh+ysn3xHQHepu1NTNE0OeUvGAdzevB8OdeZ/aR8sjgpMzQrZQilxVcl\nRHGyouHUMNThlBLNZ5zOSzR9cLa/SgavcsYydi477l0sI1xUufypqevUTmdHNzJkJV7/5x2OIj0j\nHVx0VBmysqkSTUZzY57iY1Rv1iRs+x68mmnSONk2VcVxkmWJB4asYFXFDFvaYpdM0Szcqfz9Kp3B\nUsrs+HKuM3vJxMtxHGdPDGFgExnCpTJ4gWl/GE4v0TxVKNH0axIGyw9ZqWZdnMuHz+xgDLKwqIVy\nytg57aX/jrucwevyTOc4ntxdt6zLo0jPOJUEeIfVNQkbyuCxcBnz9HGvZrYHb0sf3yiNRhUlmt3L\nh6yw6BwrSnrskveTPXdWXpNQWnSeZPiKf+Q+g3d6L83gxfkZQDNpEJT/iRedohmnS9YPhoEuj8Y6\nHMXaH1aGrJwuDlnJ1zUsm2nIRgAXXpyTwSsH8k092cexywL1vrwga9J4A2c6o3j9bMHhKNbVPsDr\nSQYvG7Kyg/cTNKONKoN1+RNX25rB28SgKPTbpiYpn2T+/NCYKZpYlVmyjFxKpmjKVCrRDINCD575\nO1n+R+4zeKfTnXWRc6XgcFjJ4C06Ec8PWTk1DLMx6dUM3jNODXU53ZHnn1T3V8ngVVLhkXNZ5vEk\nn8Us3o5N3QzFATaLBvvbJN5AgBc7t3am+XAUZbslj6YsOu/6yZ1MAuYp3u/7cj/xJZrb+mLY34w9\nuTnRA1FM0N+1/MQqGTyswQdhZpMlmqHlUzR9uWbxecsPZDidZteiuLxiYVDpwTteNINXGLLis3QT\nAd7pPUnSucsjRWlAOAxt6axbtRQsKdFMM3hbWmbThOILpqYe2KNYGpLBa1QUu7X+fZxzOhrHesap\n5O9pYtG5P5PYeQ/eZn4utkcf1yRkQ1Z6cjzLojQaVZxs656/rX3ygjUJWIkPwoo9dqUMXpA34VXX\nJPiBDD6DF8euUKJpGgbVDN6CQ1YKJZq+z26iRDMtKXv68kijOM6OdfVF5/mDmM8yneQHtDbOjrvC\nbbuLL9zznYodZvDiJOO+6pAJn7G7esqQFV/+3PVQnGxtyQ7eT9CMNvqE17XtUzRZk4Aq/9DPY3F3\nstkQ6ePJFibwCPD6wGerkgxeMkUz32VnWX/d6b1QpkoGL31xeCa9THFke92QlUVf1I+jZMjKwTDU\nucP6DJ5/QfrU5ZGiyGkYJGsZlg0cqrveyj14J/cBrTyhrrnr9ENWdjE7upEM3poN8H7JuT9hMlmi\nWX7bFSa3YR7XwmPUuo63fA8eJZqooi+ze1kGjymaWEfeJ2dZBq+46PxzrzrQr736a/UPXvh5yRqF\nmjUJWQ9enC/HXqtE0zmFaYmm/3GTUzSTkrKnLo2ynr0wCEp7/BZRrS8v78E7uQ9ocQtnxyOXZIzN\ndnNNwiZ68NYNhHwfrT9hMjFkJSvZ6vbfaxPBMrZL8emkL/eTbE1CT45nWZxYQVU2gG73nrJ7q5rB\no0QTKyll8CwZoRJVzhp8zRd9jvYGQe2ahGGYl2IWe/AsUE2J5hJDVsJkyIpX3YOXZfAuHWscxxqE\nQRaYjZZ4JPLH64/NFTJ4XT3JPXHxWHd/9nwnP2tRxbN1TZZoBpas5xjt4AsIH5R0efZ+3f4I30d7\nxcFAgdUsOt/QFM1NlLtiu7TRJ7yuvERzwweyIvqtUOUf+rlPdMff1P516RYm8Ajw+iAbsqK0RLMw\nlW9QCdACMxX/xA9HkQ4GoYJCgFcc0FLN4C3agxfHPoOXf//+sHxdVxV68MZR0rMXBstn3qpN5bHL\nb5OuArwf/72P6+vf9Mf6tp//kC4cjTv5mfOU1yQ0c50+M7tKr2Qf/OWDT+tHbr5zar9bftatu99t\n3UyXz9gdDEIdDMPJISsbOqOfl4Zu3/0E3ShWavRl0bkfirCtL4Z5MY+qTe1CPcn8bU6JJtaSZ/As\nz+AVevCKkh68QonmONb+MMhWKcSuUKKZ9sQVLVqimQ1ZKWTtDioZvCvSstALR2ON4+Ty/uctk22Y\nKNHcQAbPD5L5o088qrse7kcmr/i7N9VcHaXTUYdBsJVrEt738c/qLX98z0Sfmtf1gALnXBaIr5rB\nuOwDvGGo/UEw8bttaqpePtyFFxW77M/vf2rlx4LiXaMvDye+RHNb+7ezDF5Pbk9sXlw4+Y1uUKKJ\nRvhgxt9/kima6e6NagYvsHKJ5ijS/iAs9axlJZqW97L5O+eimY0oHbJyaq9QolnJ4PmpmePIaRyl\nJZo+g7dEBiV7gVx4EPML2rsq+xkVjrcPgc99j11sZQR57JL7RRhuZwZvXjar6zOdxZ+zzH2+6LAQ\n4JHBQ5ceOXeob/yZW/TeOx5Z6fvbKCNfVzZkZUvvt5RGo4qy3e5le/BYdI51DIp78NISTP/iKrQ5\nGbxRrINhkKWPY1cpxmVDgAAAIABJREFU0UwDJT9lc9HgJUpL+fZLPXiTd5dBYBpFcZbBC1eYfpkt\nOk+PzTmnvUG3JZrF3X3HU7JDXfnYA0/rJT/5ft3+0Lnsc03dDM65NLMbbOUZ7nn9aD7I6urFUWlM\n/KprEtKeu4NhkAR41Smabvbv3BbKgnbf+cOxnJPOp5OSl9XGIKh1jXZkyAov5uExeKd7/qb28ySM\nEk2sIivRlMkkqbLqoMgqPXhH4ySDFxZKI4srFnwvmy+nXHSwRpQOWTkoBHXVISuStBcGOo7ipAcv\nzEs0V+nBK/YeZBm8rgK8yGW31aYzeE9cOpYkPX7hOPtcU0/2vkRzENjC/Zh9Mq/fLQtKOurBK+0q\nXLcHz5doVqdoxt0GrV5Wotniz731vid02/1PtXb9mM0/1q168qDcg9fIIa1t2xedsyYBVXnZLneK\nruSJhzxhsm0I8HogW5NQM0VzMsArN7MfVjJ4UVxedO6HrPg1CgsvOk+HrJRKNGsyeMNBkGXwwiAo\nBJpLTNFML+q/Jxmy0m2AN4pind5LbqNNZ/D8C+tioNnUA3vskjLfQbj8vsI+8LfNtPtXtZ+z9eNp\nIINX6sGry+D5UsmO9xZ2UaL5IzffqX//+3e1dv2Yzb94WfXfuNSD15OIJBuysoWPb9Lmem7RXwze\n6Z5//PCvwxiygpXkGbzkP+eKQ1bKlw2s/KRazeDFcVpemX7sM2pLl2ime+2Kg1WqPXjJsac9eHGs\n4YoZvKwkJT20ZMjK8sNa1jGOXbZQftFBNK0dS/oCpRhoNlmi6dckbOOi87kZvLjbF3elHrwVf6Yf\nqrI/CHQwCGp68DYzNKKLISuXR/HGM+YnmX+sW/WxoDQIqicvPv3fU18CzmVRoomqrk9corC+K0uY\nbPJoVkOA1wO+HDGZomlyyrNwYVD+J/JrFDyfwfOBYJRO0fTpZB88nvElmosOWfEB3ow9eP76j6M4\nu3y4whTN/A8pb47vOoNXCvA2nsFLA7xiBq+pEk2XlmiGwVaeIZ63BqHrnXFNBHg+wBmGgfaH4cQU\nzbwXrtv7ZRcZvKNxtJX3w13h73urPr6UBkH15N9xtOVDVvxNSoAHj0Xn3WOKJhrhB4oUM3j+BWq1\n7reawTscRToYhlkg6Es0/bf5AS6+/HCpISuB6dResQdv8u6yFwYaRU6jKNYwCLJgdZkzwsUePJcG\nqJ0HeMUSzU1n8NLf+WjcfIAXx8p68LYxczKeE8B1XqLZSICXfN9emGTwqj14WaDV8eu9Lia3HY/j\nrSwV3hVr9+CVpmg2ckhr2/YevGxd0PY9PKMllGh2Lxuykk3RJMDDCvIMnu+xy18sBtV/ofRO5rN4\nR+NY+4PyHjyfTZOkYXoFSZ/e4oFXHCdnLIpZu9oevDDQaFyXwVuiB6/Yx5EOiel60fk4clm/4agv\nGbw2Ary0RHNbF51nUzKn3L/G2ZnObn63JlZZ+DOEg9A0HAQTJxiyISsdv+Ibd1Aaepw+dmAz1u3B\na+IER9NGa5adbhoj8VGVVXFwn+hMXg3ke/A2eTSrIcDrgeIUTSmZkukf3AeVCK+4K0/KM3j+YuOo\nXKLpM3h7g0CDMMhGvs4zjmOFlpdo7g+C2jGxgzDJBI3WmKJZvOw4ThZHB4EpsA6HrMRx1qfYlwxe\nMcBr6pDiQonmohNV+2ReBq9aN9+28gvc1a7DH6v/+6ne56vTvLrif582g2U/gRebsW4PXvH1puvJ\ni0//uLmNAZJzjhJNTPCPwc715+9s18WV593qTuptQIDXA8PSHrzkDzgr0ZwYspJm8NKPsxLNQgYv\ndsUSzeQK9gdBUk45nnxwePut9+u+xy5mH/syyeKi87rsXXLseQ/eIJ3OKK3Wg1c8/mQQSNDZGatx\n5AplrJt9AK2botnUg3rslO7Bs84zQk3IBo5M+TfqOoPXaA9eWuJc/d3yDF6398t4TjDdBDJ4m5WV\nM674WFC6//fkhefxmlnJTSoe8hYePloSlV4jbfBAThD/upQpmlhLdYqmlL+4qt6psq9XSzQLmbNy\niWaawQuDdDR+dQS7079651/oHR95oPQ5KZm06PfgFReeF+2FyQvSURRrEAYr7a8rvhj3e/wCMwVB\ntz14PltZHXLRNf+CetTCkBXfn7mrUzS73hnXxKLncZSc0AjSwHvib3RD5TnzbusmHI3jpcq50Sz/\nGNBED15fAqp8cMyGD2QFfbw9sXm7fL/45KMX9O4/f2jThzGhOrBtCxN4BHh9MKjuwXPFIKtSohn4\nHrwkq3M0jrU/DLPPR2kGLFuTkAaPe4Mg6ZervKi/PIrkXDmYyFc0lEs06wwHSYmmz+BlPXhLBA+R\nc9rzQ1XSElMzSzJ4nZVoOu0NLB0a078evOZLNLd1D165Lr5qnI327/Z4pNUDsFEcZ3+nYVhTormp\nDF7LvUD+ZNSuvWDZJsdZBm/9AK8nCbxCVrInB7SEPq6dwObFO3y/eNsHP63X/cbHNn0YJb6KTSqU\naJLBwyr2Cj14pmRNwtQhK6k4De6kJPjyvW9x7Es0J3vwhjWTEy8djyVVRvKn7y4U4KUBURS7JINX\nMxzlI596Uj/wmx+bWmYYu+T4pPLZki578MZRrEEQaBjaxtck+AeUo1ZLNIOtDPCyHrwpJxD8fber\n8tOmMnjDQsa9+u9S7QXoyrx+x3X5v7NtvB/uivWnaObv9yWg8vervhzPMooP87v2Qn6Wpy+P9OI3\nvk9/fv9Tmz6UXiq+bNu1+8XxON743IOq4k3s51bQg4eVZCWaxQxeIYtWVDyLcDRK7ngHwzD7fFai\n6ffgpRHiXhhqEAbZxD7v8nEykr344tFnQUJLMnJ7YVC7A88f+3HkkizElCmaf3TXZ/UrH/z01NLH\nZO9d8n3Feucud7WN42RIzN4g2HiA1+YUzahUotmvB9VFzMtmdZ3BGzdwZnUcFTJ4NT14/uOun9jz\n3UvtBnjb+EJ8V4zW/DfoY8Zp3d1+m1TeK7jBA+nYI+cO9eBTl/XJRy9s+lB6aZdLNMc9rOKIaioT\ntjC+I8DrA5/1MqUBnvI/4uoePP9h7JwOx0lwliw6zwM8n6UpXndSomkTJZqXfIAXT2bw/BmL/WGg\n/eG0DF5aohm5bHiHPw7vcrrXa2qA5/LF5nmAlwR5XZ3dH0dJieneYPMlmrVTNBu6GVyhRLNvD6qL\nmLfI3N9OXb24K02AXfEfaVQ4wVHXJ9v1ZFCv7dLQo7F/7Nm+++GuWLfPslhZ0JeAaptPHEQ9vD27\nkPWCbmFfeBd2OfAfpxVgfZoOWvfYUX0tvg0I8Hogz+ClJZppH51Ul8FL3jqXZ/D2B2Ee4LnyonP/\nwnEYWlZOWeQDvOPCdE3/JOODtVPDUAczMnijKM5epPrjKAaSWYBXWeCc/B7lxeb++yxdxt3VNMRx\n2gc1DPuQwUt+fjt78JIHqkGw+UB2FfOCjqiD3W1FxSfbVf+NRuM467WtXZOwoR68toesHG3xC/Fd\ncbxmiWY5g9fIIa1t3b7CTXI7XIo3Sxc7N7dZHzPlTZl30nYT6m7jujVhfUeA1wM+CAvSMZpO+Zms\naoBnytchFDN4pRLN4pCVIF+TUDdY43JNBm9cqTk+GIYzMnhJSZmf3Fk3RfMwDUTrMnj+78j34BVL\nNMPAZu7t+6HfvkP/8tdum/r1RTnnNEr7oPZqFk13rW6KZlNntyLnFATbu+h83guBLna3lX5eEz14\nsdNw4P9eJ/9G/bmSrqdN+ie5tqZ35jvYtu9Ew67wa3NW7VntYw+eP0nYl7UNyzip4/BH2TRXHgvq\nlP7OtvB+PctoQycwZ6k7FEo0sZJSD54kufzFVbWx059EcCpn8LIhK276ovP6DF4yZGVUN2QlvY5r\nrtjTs87sTT324yhO1iQEQW0Pns/gHdZk8PyDlQ9y84lFyedmlWx84pHzuuvh81O/vqhsYmmY7Arc\nfAYvLdEsTjZt6MHPpQN4BjXlun3xxMVjfc+NH9WFo/HE1/JsVv2/URR3e/a+kSmaUZz1yoZBUJqi\nKxV74dY40BW0fWZ1m0vpdsW6mZPqDtNNK05l3cb71S73Ws0yzk72nJzfeRlxzfPBrqjb+7tptSWa\nWxjhDTZ9AMhXGUgmMyv14A0mArx0TUKsKT14qpRoFtYk1JTl+eBrVDNkxf/sn33V38omfVbtheU1\nCcOaKZqHx9N78OIswEuu3wc1YZCUlM46ozeKmtmhNc4CvP724DX1mJ6XaPY3g/fRTz+p37rtIf3T\nr/1CXX/ds0pfy7Ob0zJ43ZZ7lHsjVszgRS47EePfjuNYYZCURc9bDdGWKAss2w3w+lSac9I0uSah\nDwFe+UTl5o9nWeW1E9t3/Kva1GPcttjlzG6Wce/RL1b32EGJJlay54espBk851xhXcCUHjy5LCN2\nUNiDF8flRefXfc4ZvfJFz9Xf+eJrNBxMZsR8D15dBs9f5+dedaBnzsjgjcZx+iK1mMGb7MGry+D5\nn1Ut0bR0imaxN7BqHLlGzvj5nzkMgiwjuUl1GbzGSjTjZKn2IOzvmoRZQdz8Hrw0KNnEkJWVSzTL\nPXjV680Wne/ckBUyeJvmSzRX78HL3+/Da/PjrMR/O0vZyj29mzuOro1mPOajktndwvv1LFEP/+3r\nbuMtTOAR4PWBf3FXnKLpzyBM9uAlYlcs0Qyycspx7LJl1lISOP3oN3+FPveqg9rBGpdq1iRUh6zM\nMhwky9PH6ZqE+h686Rm8qJLBK07R9NnBaZLhLuu/qshLNG3tEs0LR+O1e4r8v4U/jkX75T507xP6\nrdsenHmZ2DkF6bTTvp4tHc/ox8i/NiXA6zgYqiulXNYoyqdo1g0pilsOtKZpO8ArZvBOUraiT7K1\nIiu+uOrbC0+/9uHUMOxFwLmsk16ieZJ+52XsconmqIf/9nUniLexRJMArweGg+IUzcoevGoGL72T\nudKQlTBbiB65coBX+jk1fVeXa3rwosqQlZnHHpiOo1ixSwKk+gxecn21Gbz099ybCPAWKdFcPYP3\nc3/0Sf2XP7kvux4p7cEbJHv9VvWy/+eP9Z8/cO/K3y9NTtEcBLbQ2dxf/O/36U3v/cTMyziX7hgM\ngpVf1LVtPKMmf24PXsflHk304PkJrlJ+oqPuejtfk9D6kJX88aBHz+0nytqLzgvf14cg3T+Wn9oL\nexFwLmuXpyXOkg1Z6VEfVp8Un6p37X4x64TuptQdSt1r6r4jwOuBYVDcg2dyctlC6okhK+nb2OXT\nKYsZvGqJZunnzFiTUA7wkreL7P0YFnrzBsU9eIXrm5XB8y8Q8kXnvjQ1CRhHs0o043ilfrl3fuQB\n/ejvfly/8WcPZtfjj3+dNQnOOT3w5GU99NTllb7fy3rwomKAN/9B/WgczQ148xLN2RNKN8nfB+pK\nNuZO0ew6g9fAGfdRuoNRUu2Qol3P4En9enI/SXwJ/KovGsv3/0YOaS3++WB/EPZut9Yi3A6/kJ+F\nNQmz7XJmN+u/7NEJ57qTQ1sY3xHg9YHvb7vyYJBn8GJXG2D5M/3jOM4WBR8Mw1JpZOzqs291fVf5\novNiL1FeGjiPzz766w/DmgzecX0P3o0f+rQeTIOhaommpRm8WUHIOHJL123/6T2P63W/+TFJhRIx\nn8ELTPtrDFk5HMVyTmv38Pnj8eVGgzBYMMCL5/5sX6LZ5zUJsyaqLdqD19XZ+7iBM+7jKM7u/z7Q\nK/7u/m9p9xadNz8lFstZO4NX+LY+ZMz8feogXeuzbXerZYdp/NUj5/XWW9arGOmDPu5C65Mmnmf6\natTD4L6uDJZF51jJ1zz/WXrP//0/6YvOXpHswXN+X9nkHco/cR2O4iyDdzDISzTjrERz8ucMg8me\ntsvZovOaNQmLBHhTMnjFP1ZfSlp8QXfhaKzX/sbH9K6PJlm0Yc0evL2ajGPRKI6XKum4+7MX9B2/\neKue96zT+jtf/DlZMJQNWQkDDUNbOYPnV07UZSqXUc3gDUNbqJ/kaDT/9vAlmsl9oZ9nuEfZk31N\nD96CUzQ724NXPDGy4hnIcVycollTotnx71T9uV0EeH16cj9Jsh68Vffg9a5EM+3B2ytPoN0Wy04l\nfddtD+oH331HL277deQlmtv9e7Sl/HywwQNpQR8nqNb97VGiiZWYmf76512ZvC9f4uhqh5wcDJIn\nrsNRlGXw9ktrEtzU7F9tieZoMoO3zJAVPwHUX754HF5dBu8wm6wZp9fjA7wlSjQjlwUDi3jvHY/o\n/NFYv/Dt/4OefeX+xJj2ddckXKoJlleR74VJjitcokRzXkYzSoP/MOjvGW4fpNbdjvN68LrOdjUx\nJj4p0axk8GrO2HY9jCRbdN5FiSYv7DbCl2iuevv3rXTMP3b758lty3b4v+8wsIVO6OQLwrfr96zK\nqjZ69CK/T/q2jqRJfQzu6x7LtjC+I8DrG7PkQT5y9UHawTAP8Io9eEF1imZtiebkmoS1h6wUMnhh\nGGQvVP3PiWOXnan3x+uPv/h22pCVWSWayZCVxZ8Q/AvKL7j6lPYHYfbx/8/eewdIcpXX4qdSV4cJ\nOzObd5V2V1lCAgECREZGRMOzxc/C4GdsjH/YYIMNxhj8eJhnwM+BZDIYGxBYIJIAESUhkrTKK22Q\ntDmn2ZnZCR0r3PfHre/Wreqq7uo0XbP0+WdCp6rqqrr3u+d851ii180zWWmzQCs36DVsBeHBWleT\nSzSbFaeud17JeWv9gOMyvPvbW7FncqHusUZOmc0KOBEKvmgxCdG/twLLcetcNO3A9diabKtboGPc\nM5OVhD14C1Ub7/r2VsxXrJ5sx28yOnWwS1s+lyjwjKXJ4NFln9RYK9xmsFRhLfLC3FJDQAp9hh0j\nx+3sHtQLhOcPqjLIwRugC1DgxyREFVimJNGsWg5MXYWi+MyZ6zK4bnRmR1TGW2RMQpsmK4bH4CmK\nf9GSPBOAYBxp+/njngxR90xWbOrBQ3OJpufeGZYJ3b9/OpLpsBwXmreN3C0zODgampIoB+/gVAmH\npkt1/+8egxfcdkNLNthXbbfhAMkYA2NexmBEr9diYnK+iq/ccxC/2nWq7jGhyW/kohmz3aIo6YfJ\nShd68IyIHlZ5VxazIBfFcq8YvJgiNoyHD53GV+85iIcPzfZkO36T0am5RcAUJAUTtHAPXhr6AluB\nK6lnktzD6Pvrd3Zrp3Cc+Hv+AGe2u2oqXTRDh3gpyjOBHhd4iqK8SFGUxxVF2a0oyjsjHn+2oigP\nKopiK4pyfS+3ZalAkXrwIiWaxODZDqq2C9PrXaNizCH2L9JFM1nQOd1MkvTg6ZJEk57PM9b4ewRZ\nO/93KvaqHutlCPMYkmgqCSSaHgMn3RjuPzCD6z99Nx4+XD8ZlJkSmanzJZrJGLy//voWvOeWbXX/\npx68Tgu8OgYvqcmK5XCTnTh2y/u36oXIR33WYoHOt8goBMoBbGCyEsvgLXoOXr3bZauwHL8HT4vI\nkQxm7bX1EW2h13LXqnQ/aLTQEGbaB+ge6P7aNoPnMiFdSsPEk+ReNE6moehsBY40FiWRY4ug+iV+\nbfTLSGqpwE0ZU95NiO8+RSx0+H44KPBCUBRFA/AJAC8GcAmAVyuKcknoaQcBvA7AV3u1HUsNCvyY\nhEgGzyvoqpaDiuWIgUz1mDPXjc/B0yNy5codFngZmcHzfpcdGstWMgaP3odWIlUVzXPwIm4MUws1\nAMB0sVr3/JrElGR0VWyPLSSaXtC54zYcXPeeKmK+Ytf9XxjWdDjYhm8uesJ+DFq9jpO10iChqbJb\nY38mBn6BF8G0uvGTlmbGH4vO4Emb2O7kxHZdGKEevKjrkZ67WOh5TIKUg9foM+h6WuosRRohVAxt\nu2gyce6mgS2Tg86B5Ofu+763A/9++66ebVdS0CE0NCXR8bTc+PvoUkIa+7DShLT1unYTdof3oF6g\nrsBbolrHXm72UwHsZoztZYzVANwE4BXyExhj+xljjwAYjNweBIMXY5RCBV3VdjmDZ0g9cAofFFw3\nLuhcrXNOLFm8UAlINKVCoBkCPXiCwfPjGKjoARIyeMJkpXkmXZSdPrFoC9X6UHXLcUUhmfGOhesy\nn8HzCjw6/lEoVm1MF2uRbAKxoXIh2w7qGbzkEk2gucOkIvXg9Wuw8N0wI5wyG9zwRQh6RKHjukxM\nkPoSk9BugefILpr130u/Bnf63F4xM7WELpoDBq938Hvw2nfRpHM2BfVdfQ9ewo26e+8U7t0/3bPt\nSgpfoqkmuueLSJ0lfm3Y4jxMwUmUQpzREs0Usrf1PXgDBi+MdQAOSX8f9v43QAMoCu/Bc9xoBi1o\nsuIItzCAs3iOi3iJZoRDXxTrRIO9lmDZIpCDJ4U10w2pEsPgVcMMXkRMgqEpscUKZyr57/Jkv+jt\nT6laz7BZNgsweADfb2Gyoqlif+LYgkMzJe/x+u2KipxoB+HJVnKTlSAjGQa9hSr14LXiQtpN+OYA\nEQVeg+KvUQ9eN0LHW0U3evAsiVnWIq7ROLlmr9HrgVe+ThoyeIMCr2egAqHd88pl/n0/DZNzum+3\nGpNgOW7H5ljdgHCw1pRkEs0GUvelhEb3/AGC0vylJjtuBl+imZ7vng4x3dsGBV4PoSjKnyqKcr+i\nKPdPTk72e3N6DMVjkNzoAk/3TVbKliMGMsBj8FzXM1mJKPD0IEsGRAedt2ayIsUkSGHNxLQECrwI\nF80wg1cLFHjxEk25qAvsT5UYvIgCz3GFmYspFXKyyYpw84zp/Ts0XRbvFUbXevBCxYuhNW+4d1w/\n9D22KJaYWdHr1SdJDB2/qEK50aSlUdHRj0KoG59pu36/bdiFFugjg7eIJisNpdgNYjMG6AzdcNGk\ne3camAU6R2icTLpftuOm4vxigsFLZrJyxsQkpNBJMU1wpUX7NEihu4k0SzRJnZCgWymV6GWBdwTA\nWdLf673/tQzG2GcZY09mjD15xYoVXdm4tILXVAwOa87glaoO8nKB5zF4cUHnNImkgYwxJnrkZHOO\ndnvwZAaPJqhxPXi0WkrbQoUiFVaqEi0pJcgTYLkQEAxerV4mGe7Bo8+nwYViEgCg6kTLLA967pmR\nBZ7VGxdNTVWayp/kz4xbBaWBgdhRIL5fr9dotGpnx/RjBCSYTQq8xVrlDDS/dyTR9BZHIuIrbJeJ\n83IxB0E6XxbDZCVZD156JgBnCjruwXNZwMG536CCxxQmK8lflwoGz9sEbqzV/PlnyuKHkJqm4BxK\nI/hCCl1nfd6YLmOx++aTgMZ1mi8miQxLI3pZ4N0H4HxFUc5TFCUD4AYA3+3h550RUMCldPLAKcMv\n8FyULBv5jC4eo0BsJ+a19NyyFDLOGDBs8v9boVW0JAWe3IMn+ogkF02SLZq62jgHLyTRVBRF+l88\nyxP+nRi8YgyDJ/fgAXxgtKIYvJjJJMUjWBEDaln04HXXRdNI4KIpF89xjerM2yw5VqNfN1WrgezO\njjEOkI9LJIPXBblkq3CabFMSWK7v7hold3NcBlNrjZHoBojd7dXEveoEi9g4CInmEp/EphGdSzR5\nv7eWMLet12i3B6/muKh12DvdDcgxCUmuu05jLtIC32RlcI1HwXV9M6M0MOXdRBrluXQ/FAXeQKIZ\nBGPMBvBmAD8G8CiArzPGtiuK8j5FUX4bABRFeYqiKIcBvArAZxRF2d6r7VkqoB4823UjJZKaytmX\nih3H4MUHnRdM/tyiJyUkSeFIzgAgDfYtmKxExSRoWr2L5rK8EcngVQWDRwwFSTQlp8eIJSt58i8P\nbsTg0T6GX0OfQ+Y0NdsNWFPLzF4UDosevHiTlW4zeLra3FFNLirj+geFRFOJlgIuJmilNlqiGX3D\nDxY9Ef15Dot8bi/RafO747GS9H3QNSSf367E4C1qgceC94NG2HZkFjd89u6AJLsZEvfgnSF9RmmE\n1SGDR73iqpIO6RidU626aFqOmwqXVpk5SHI/OVMKozOlUO0VXCYZcKXgOusm0miwIy+0AIMCLxKM\nsR8wxi5gjG1kjL3f+997GGPf9X6/jzG2njFWYIxNMMYu7eX2LAUo4M3VjhtPC2d1jUs0a8ECTyUX\nTRZ9QhKDV6oGZYxU4IUd1ZKYrETFJMgumiTDGstnGjN4glGTZYTx/XBy0Rdg8GrE4EVING0pB0/j\nxy1gsqL6nxlXpFEPXtTjwkWzw8E2Kgev2T09kCkWo+EQEk1VaVg8LwaIjYmWaNJgH3wsLFsMoy8m\nKx324PkGP/z7MCKYOof1SaLZgnTmoUOnsXnvNCbn6+NJ4hBw0VykHLxi1cYbv/wAjs9WOn6vMwG1\nDidXjDGoKr9fp0GiWRMMXmtsh2WnowePbnFJnZP9Xub+b3snSKPRRprguFKvawqus25BNstLU0QG\nXYc+g9fHjekAS8Jk5TcJMgsXx6CZhoaq7aJYC0s0+QXDIxbqX1fIBBk8YtdGsp5EU8h1vPdLZLIS\nFZOgiCKRPmM014TBC7toqmjoaCnfDOTfqbCLkmjWHFdMlIM9eMTgKXVSURmMMakHr/5mVJZMVpI4\noMUhzE4ZmtKSRDPOIIbeQ5ViEtIYdO5LNoLbFpBDNghBz+jqknHRpP2nBQ4tovB23ejCr9dopcCz\nQtdzElRtR4RkJ3HR7MYEfNfJBfxo+3E8dHCm4/c6EyAYvDYn1o4n0VSV5veoxUCdRDMxg8fSUeDJ\nJisJjmdcv/JSQxpZnDTBDZgZ9XljuohmbRf9giOY9AGDN0AXYeoqql7REceg8X42B+UQg6cpvPct\nXqLJCzkqfohxGg0xeLRClCTcUY5JMCSZWdhkZVneiGTw6KLOhMwlFEWRYh2iJJrRjJVg8CIlmvUm\nK1XbkYLOJZOViMF+ulhD2XJQyGiRhYlsKNPJimr4RqepzQsWeXvjjFOCMQnpkGhGFcpx2U5x0QHh\n/5na4hV4dK0YmtKWIyntqx7qwQvnUpr9lGgm+ExhhNLCJLlmu0JKl8hFswvnaq2NQvRMRjdiEjSv\nBy8N5EvNdqEGs7feAAAgAElEQVRLCoUk+8UYg+Wmg8ETMQmq+psVk9DEAfo3HS5jfc+u7QWatV30\nC/64Hlx4XWoYFHgpA7FzjutGsnAAl5/MV2zYLhNFG8Cld64bH3Tu9+CRRDO6B88P/k6QgxfRg6fL\nPXi1IIP3o23HcNuOE4Fij79PqxJN/3/yoCB68BIEnQN8oiebrAh3yYgBc7pYAwCsXZbjhXToRis7\nd3YyWQjfwI0ELppJTFbofVUl2q1xMZHEZCW8H0FDk4gePJnBWzSTFf7T0Nr7TCrGfRfNCIlmH3rw\nmjmWhtGOjLLmuEKBsFg5ePQe1RYMNT70k8dx155THX922uC6zHewa/N6cV0GReG942lh8AxNbclE\nivpg0yBzFDEJCaJxALnA6/+x7wS04HcmFS/dRECimYLrrFuwAq026dkvwaR7x3yJEnjQmz9lgMUE\n9Q6Ua07sqkHW0ESxQSvggC/rcBiLlFf6PXi2+AzAZ/BEP4bo1Wq+vcEePIpJ8HvwKrYDU1eRz+io\nWC4+evtuDJs6zl81FHgfUcyJHLzGEs0AgxeRgxfpohkVdC7HJGiqn48XUaDNli0AwMRQBrtO8puT\nqfrHv9ylAi9c2JA7aiPIPXhxE2Eh0ZR78Pqcgxe1rXEmK0kZvIyuYn6RWqx8KYfaVm+EyGBUQwye\n917hlcTFKshblZ4KGWUrBZ7tCgVCQxfNLrIU7TB4n/nFXsyWLTxj4/KOPz9NkCdXnbhoaqqS6B61\nGLAcvhhCCpYk566cH+q60eqXxYLowUvoSppGB8J2QOxNv2J70g6XASbFJKTgOusW+mGMlgRiLjGQ\naA7QTWR1j2WrNi7wZrwCj1g5gE/cHa9ptaFEM5QVN5L1GDw3KNFMxuDF9eB5BV7NQdbQhKx0cr6C\n+aody+D5LprNJJoSgxeQaMbn4PGgc89FUyrk6L1kk5WoAXOuQgWeKV4rI8DgdTDgyjc6RfH7Mhsh\nINGM+WxZotn3mIRGEs0YRzV5MIgqCGypGFqsQojiTLSEPTNh+BLN6B48es/FZvBaNY9pV6JJBV5j\nBq97k1jR+2sley/bcVG10+Gw2G3QuZc11IYFdiM4brp68CjrlBY4kyy6yN9tv79nJ8QcNJNpCrOq\nJV4YWR1Khc90yD14Z9IxCjB4KTqHwwzeQKI5QFdA9v3Fmt2gwFMxRQyebLLiDbJcoln/OppMFUMM\n3kiOv0dYopnEZEWOSZD1yjTglC0HOUMT0tOpYg0LVatOIqVrChRFDjpvItGMYfCo924hxmRFuGhK\n7KBgUZrEJMyV+Xuu8Aq8cHES6MHrhMGTbuCaokBRmq/mBkxWEkg0GxWyi4FGEs14Bq8x4+BKxZDL\nmk+OugFiy1WlvR4kGtTovDRCvZGir7BPBV5Sw5p2GLyq7SLXAoPXjb45PzQ92XuVvGs6aUG4lCAb\nklD2aqtwGUSBl4YauGa7yGitLWDJ95l+92YyoQhItv1ioSzGWGupQDgnp0imlyZwieaZx+AFes1T\n9N3TZUfHfIkSeIMCL20gBq9Uc2Jp4ayuCblgISIHz2HRQedUwIRz8MISzZZMVtRoBs83WeGTOJqg\nMgYsVOoZPG78oYgJb3OJpsTmBGISgv2FwddEB507riuYMvH/RgxeISPeT0apZovQ+E4mCnKzMWfb\nmhcr8ufFOeKJHDy1/wyeH2Ye0YMXI8mTB7bo15GsItptrFi1sevEfPsbHQHH5Tbxmtpek7hg8Mig\nKDSx86Uii1zgMf9zkxTLnTN48a+jAOpu9GgIiWbCvD6KlOk0+iSNoO/MN7ppp8CjmITFWVBpBlJp\n0NjZihMl0B2n1k5Ap5muRt/DwhBS9xSxH+1AxCQs8f3oFVyXCTbpTDpE8nhm9WkuEoVB0PkAPQHZ\nOxertujHiXsOALECDkCsosaZrAC8IBQ5eFZIotmGyYrcz6WLHjzF78GzuERT3uaFql3H4KlegUUD\nrKpKEs0GRhyALPfjTmimrsJyWN1nyEHnMlNnuUzsQ6McvNlSc4nmaN6IfX1SyBMtkTHVSg9ezI2S\n3kNRfDOZ/sUkNJJoeudh6LFmPXhuSM4Ynix86e4DeMUnft3ViSiPJOGupG0xeKEcPDoPrZBE05cw\nL873RYs8SaWh7UQZ8AIvqB6IgjhXujD5tlpkA2kx7Mxk8PhxbTUUXIbrMdhJZOSLAVrEowWsJJNh\nebGo3xJNXxqWjK2xu3ht9BNnStxDr+Awn8E7k4LO5WsvVS6aoaDzJGq2NGJQ4KUMZLJStd1YiSbJ\nOAGgEMjB4/lzJJuJQsHU/Ry8Gs+hGhI5eB6D14LJChAMOOc/pR48y0HW8M1L+OcwzHkMpLztGU0V\n8Qkyg9fISh/wBzdi71aO8AKsFHLStGw/JsEMSDRdse1mA9ZwrmIhZ2ii7zHMIpVrjmBDO2PwpAJP\nyJ+aMXhyDl4cg0fv6YfY92vFtJEzYqxEM2EPHhUl4V07XaqhVHO6KsNyPFMGVW1POuP3DQYLPJKr\n1BVaizS4+/ElyT63lQKvYjk4PFNC1Unag9c/kxW6h/R74t8L0MKZmSCqIg6Oy6CIHryubl5bqHlG\nWkKhkMQgSC7w+lwo0T3EEAxeE4kmSRvTcPA7QFzf9QAcLvPnWWdS0HnQGTs9+xVm8JZofTco8NIG\nmemKK9JM3X9OPizRZB6rEPPNFjJ6IAcvZ2h14d50cidh8ABpcip++k37Ze8z5P0CgFMLtcDfqqLA\nNDTRF6gojQ1PonLwSJZJPXLhPrya48LQQz14nskKbbvf9xfdgzeS06XtCkp7bJdhWZcZPOrvah6T\nUH88whASTaX/LprJJJrBbWtm/FHXrxY6aNUWJ/dJIFwEExThUZAzGAHZZCWmB2+Rvi/h3qknY0Ja\nKcJu3HwAv/WhXyR30Wyjv4/AGMPmvVOCtfULvGQSTZ/BSx6rsFRA31XOaF/+yxg/Z9td4Og2ap5E\n02fwWu3B6+/3LO7RSXvwWuwpTSssweAt7f3oFVwpJuHMYvDkVpv07JfowdMHEs0BugiZ6WpkskLI\nm0EGz/V68OKsnvOmFnCazGc0sVpohSSaSY2DqFgSUk2PSQQkkxU9eKqdWqgG/lYVBVlDFbJRVSpC\norPS5H4s/jtl360czor9kxHowZMCzW23PgA9jsEbzRmRhScVpsvyGe99258oyIOcovDvoalEUypa\n4gKhabKgKIqUg5c+iSZtZ7hQlRm6qMEgvOoWnhzRd9LNSRxJNNU2XTT9DEZaKVQCJkVhF83F+r7C\nDF4zdqcV85JTCzVhSJQzEuTgtdHfR7hn3zRu+OxmbD86F3ivpJJLWjRa6hPoKJA7aRKjmzg4nqFX\nuwsc3YZFJitK8vubbFDSdwbP+3hqT2i0+eSYDcRPjg9Nl/BnNz4glDFpxYDBawxZonkmHaJW3ZoX\nC2KBkySaAxfNAboBmelqFJNAyMvP9wZZxhr14OmSi6aNXEYTq/RUWJALp5Jw1cKXZsoumh6DF9GD\nB9SzKJqqwNR9Bk9VIDGL9Rd+MAcvxOAN1zN4NBiKQk4KOred+h68qM+cLVsYyRriRivvQ8nin7Us\n1xmD53rbSdvHV8ebFw8ywxC3CurHJPjfVf8KPG9Aj2JnmwSdmzHOjs0MSWhS381+KtdbTNEUpS12\njY6DIbnR6tL1IyZ8i22yIoppLbAdcWhFoikX2CR3TsLgtSPRJDOq017/bKssLi0anYk9eDSpph68\ndqRfDktXTILluF4OHv+7lYgPoP8FXisxCfL1EHdt3LtvGj/cdhz7p4pd3MruI9z/P0AQrst8450z\n6BgFg87Tc48NX4dLtL4bFHhpg8zOxRZ4skQzkIPHT0xiFaJQCDN4hi5uHGTOYUs3kySgAjHI4PH3\nKlUdFMx6Bi8MVeH7ThMvtalEs1677TN4Xg+e5KTpT6R9piSjqUKiSf+PKt4IcxULIzlDFBDydpUE\ngxd0JG0VdGOhPks1YX9L0EUz+smyi6Yv0ezPTdWXFsX3V4aPIU1ITV2LZJRka3/5b0K1RXleEjie\nQU/bOXgumaz414eu+sWiOB8WuQevmWFNGK0wePK5mkviohkj2U0CYi6qwomzxR48kmjaDk7OVfDc\nf/kZdp9caHk70gg6FmYHLpq0mKimJujcDfTgJdkmO0UFHhMTy+YSzSgVSxjlJRLzIUvS0+DGmjbI\ni9NpYro6RVoZvPoevKVZ4Q0KvJRB7q+LK9KoCJRt/envRkHnAGfwiNkqWw5n8LzBhPrOyPo6KQxN\n5VJCQWf7ErqFqo2CqQsGb7nXHxeG6jF4BEVBY4lmYPXSKyZDDF5RYvBqEUxJRle93jlXDKiKooj/\nhzFXtjGS1SXzlwiJZo5LNNudKPgslSa2J4kFedV2xXkRN8n2c/CUxD0evQLJohqF2Ddi8KImo2E5\nY3hyR5P8cERHJ7A9x1qSR7cK2lfZMVcLMHjhonVxJmr0+WYPTFbkySaZRDUqLoQVfBuLEVTI0Xfe\ncg+eZLKy91QR+6dKuHffdMvbkUaQRJMWDNu5F/B+b36PSoMJXtUOBp0ny8Hzn9PvOAzR+5MgJsFO\nwOD5Cxwp+HIaIJBrm6KJflpwpubgWSn93sN5lAOJ5gBdgZmEwfOKpXxGC6ws8MKKGLDo94/swQv1\n2TRiAKOQ0dRAHh4xeIwxFL1sOGIgNq4oSPvhv4Z68OS/G0o0Zfclb5+LIRfNouSiSRO7jMQkZnQV\nNcfxGEup8NPUyAGzFQav3QGVbnJ+EY/ELppDZmPLeXoLRZECtftV4JHJSsRx8vsxonvwTCOuBy/I\n0ob3rRcMnutNcNtm8EI9ePS7fC3Kjy9WI3q7JitJJZp0Pxj3MiUbyVs7MVkJf+ctu2hKMQk0WU67\n3C0phEQz0/69wGXUJ9ze+d9t+BLN5JNhK0UMnjA4SzCZl6+HOCWGv8CR7h68tJptpAUuY4mjM5YS\nnIh5XBoQNhpcovXdoMBLG5L14PGvTXbQBABN8W+UsRLNkItmPqOJGwfJ5WhVNilkSQzAHcBs10Wp\n5oAxBBi8DSuGxPOGTEPa9iCD10yiGVi9FHJQvl9kslJsINEEICSackwCwBmiiuVgaqGKl3z0lzgw\nVQRjPNpBNlmpSc35NBEkk5W2GTyHCjx+LHz5U+PXVb1AeVWJX81lkkRTuDX2TaLJAj9lCIlm6BjS\nsTF1LaYHD97j0b0KvejBc5jXJ9mui6aQaAYZPN9oprHstFeQg87l7YxDrQHL9sOtx/D339kq/q7a\nLjauGMItb7oG1126mr8uiYtmG9cU9abWM3gJe/BqPoNHk+R9p86MAo++K5/Ba/34kossd/rt/8TT\nchjPwWuBwZMLpX4zXT5z0DwmIRDQHlMULRkGTzr3BmHn9XCZ5KJ5Bh2eZtFH/QJtFi1wDiSaA3QF\ncn9dnMySJv9yBh7AJ4Y0aMdKNE3dMxZxPZMV3Z/ESTEJrRV4SmCCqqsKLIcJKWjB1EUjv8zgDWf9\n7VfVMKPnT3qjC7x4Bs+XaMrZcMEJKyBJNKWYBID3BZUtB3smi9hxbA47js5hoWrDZTwUPqPXbxdJ\nNEc7NFnx+8zkHjz+WKMJVNV2YeqcjbViBkhZoil68Pp0UxVRCK5bt19x2U6CwYuTaLpBljbeRbN7\nIyQZEsl9p61AuGiGGHD6vxsqtBYtB88JyV07YPDufHwSN24+KAxPKpYD01BxxVnLkNHVgOtu5Ht3\nQaIpGDzhopmM0aBFI87g8dfuP0MKvG64aLrMlyinoYeG9+D5C1jJJJrpY/D8HsL450YZjYVREQsc\n6WbwgkxO/8+jtIFLNJNlIy42JuergXaYVmB38Xv/wy/ciw//dGdH70HwXTQHDN4AXYSh+RN6Peas\nosl/LsTgqYpU4MWsOBDrV6w5nsmKJgweRA4eYyIIO9k2q4FtLZicJZyv8It+OKvjrPEc3vvyS/Cq\nq84Szw0UeGEGT+ovjJZo8m1VFP8mQZOxsXwGqhLTg6cHmbqa48JyWcDkIp/hbp7EyhVrDua8fZFz\n8OTJAEk0xwsdmqxIRQzAC181wWo0yd4MLVq+CAQlmqrXN9OvwZTONcbq94u+z7gevKyhRhYEdMgz\nTSSa3ZzsiB6kNie4IgdPXiDRfJfQZsYxvYIbYhIS9+BFnPcVr7jafmQWAC1GBKXoyVw0W9/3SpjB\nazFyQWbwyLDiwFQpFcVMpxAMHpmstHF8HddfhErDIanZ7ZisSExYv2MSvE2hMbJRX698PcQtfpSX\nCINnOf6icpqYnLTAZcnOiX7g9z+3ue3CSm4p6vR7f+z4HHadnO/oPQjh8W/QgzdAV6AoSkCeFwVT\n6sGToUkr/7FB516fVqlm8xBy2WSFJJpOfFB6FAxNDRRIozkDtsswOc+z7goZHYqi4HXXnIfRvCEK\nu6FQhl+4B6+hiyatPhuaeLxYc5DRVGR0lUtRIySamZDJStXibKahygyejlLNEaxcqWZjzmMfeExC\nVA8eFYBeD16bRYTfg8e/W01REq3m0qRZ15TmEk2lPpB+sVELTE7835nnAgvUs3tBF80IaWeIwas3\nWWlNnpcEZBOvtWkTT9LEMAMeDjoXbpaLVJDXS0OTSTRl2TKBrqNHAgWef++SXUPDYIz5DF47Es1Q\nUd9uD57jMrFgVHNcHD1dbnlb0oZwTEJ7Qed8rEiLi2bN68FrpVgIxiSkI+g8kURTtpiP2U9a2Oh3\ngHsz2I6LbELH3t80CDYppUHnJ+YqODlfbf7ECMjznU6/94rlirGmU/gxCfw+Mgg6H6BroNXtZjEJ\n+ZBEU1UVMVg1ZfCqNkpWMOicJo8Oa81kxdCUAINHMkWaBA1lg9tJf8sMnqIg1IPH919RouUntutC\nUfiNgQqaUs0WsRHDWR1z5SY9eB6DVyfRNFSPweM3i2LV8Qu8mKBzWukfMnVeOHaLwVP4MQAaD/ZV\nS5JoxgWde68n+a6uKn3rwQv2UMr9F35hEWb3Ai6aEfsYtvavz8HrftB5wGSlAwZPlmhq0vdC+2TG\nFK2W4/akJ8wNnYfNThOrIYPH//fI4dMA+PeQlMHjZk2Ife9mqIYYjJZ78CSZN0lMgTPDaIUWybId\nxCQ4bgpz8DRVjH9J2I6ARLPvLprBiWWje4oc0B63+BFmsNMKy2UdMclnMkSxkWChtx+oWL66oVVQ\nUZc1ovvqW0HZcrp2nvtF9aAHb4Aug250cRLNOJMVXW0u0STW7HTJguMy5DOakOsJiabLhI1+EnAG\nz3/+SDZU4JmhAs8zV6kzWQkxePTeVcfFLVuOBKR1lsNgqKpXpPCLsVh1RF/ixJCJqWJVen60yUrV\ndmG5buD/+YyOksULYIAXjjS5G80ZooCQWahS1eYFp67B1KJjFpLA7zOjmAT/WDSaQFVs3tdkqPEM\nHt0/6f2aSeN6iUBIb0SGXy5i0ikKvJjBIOxIGZ+D110Gzw+jb/31tM1hiWYdgxcjO71ly1Fc9+Ff\nBIqPbqDOvbPJ6mpVMHj1A32FGLzDs95zXKFCAIKS1DBowk2LMYwxnJyr4Mr3/QQPHZxpuh804Ncz\neMkmJPKKcKDAOwP68OjYkotmOwUaj9ThDHYaZKuUadpSD550P0iLRDNJTELUwlgYS4XBc+QCLwXn\nUZrgSguzPI4kPcfHdtyAAVXrr/cWmWIWbZPCcRlqdvuFZhjiOiSJ5tKs7wYFXhohJJpNYxJCJiuK\nX+zEsX/0GpJP5ry/ZXOOVmMS8hktYA4jGLzZ6AJv2Kxn8LSIHDyAT2wfPz6Pt9y0BT/eflw8bjs8\nu05mrEo1WxS9E0MZTC3UxPNJOlbH4NmuCKsm5LwevDL14FWlHrxsdExC0etnVFVeqLafg+fJEKWs\nQ1UwePGv4wyeCkNXY1k5MVB47yfb8S82rBiJJp2D9D3KxziYg1e/3U0ZvB724DUzColDLWLhQTZs\nabZPJ+YqqDkuTpdq6CbChWWjXWOMNTRZoUH38EwZUwvVlnrw6P2GTF0wutuOzuJ0ycKeyeZFVthY\nR5isJO7B81UAs2ULWUNFztCw71Qp0evTDLpP5DpgTlxGTr/9ZxYclwkzCuGimWCbAjl4KTFZSWKJ\nLxemzXLw0szgkSyfFq4XK+tzqYAOhzAzSgFTTqh0OKZ2S6LpZ9x2SaIprkNfSbUUMSjwUggh0YwN\nOo/uwQtINGNdNPlrTi1UA+9haKqQfDjeqmxSvOXaC/DP1z9B/D2S44XbkdMV7zOjJZpy4adE5OAB\nfKA7PsvfZ6boT2Ipu073IhkAr8jy3nOiYGJqoZ7BIwdMwDNZsV2vwVti8AwtINEM9ODl9LpgeP4c\n/7MzmorTJQv/+P0dwkk0KcQNT5diEhIweNxkReMsbsxMi1b+ZAavb0HnMSG9YkXPoAJP7sHzCzyX\n1a9kNgvnFpP9bsYkeIshPCah9ddH5eDJbrj0nnE5eNQX1up51gzh0PhGk4q4Yp1QsRyx6PPY8flA\nDh6AhsUx3c/ovmU5DHu9wq5ca77PgrX1Bn7L8QvRJLb+pRCDlzM0rFmWxYm5StPXph30XZkd9OC5\njEHzVAb9ZhYsie2l23mSbaJzTFH6z+AxFpxYNu7B448pnvrm8ePzdXLtSg9k6d2G1eCeP4B/76VM\n3DRIoQmkcGhfounfgzphbmk7ulXgcXdgfw6+VCWaevOnDLDYaM7geRJNM5yDJ0s0o99bMHgeu+UX\neH6h5DgsVh4ahfOWFwD48QfhHrzhcA9eAgZPlmhSMUqunADZYQclmqWqjYK3P8uHMzhVrIExBkU6\nLlE9eAp8rTXAGbyS5ZuscBdNS2w79QbK/Rqlmv/ZGV3FnY+fRLHm4JpNy/G8i1YmOIoctph0yTEJ\nzftJiBXhhXpjiSaxu4Z07BYb8QVe2NkvisHzJqSMQYX/vdHx8a39g/JOmjh0VaLpehI1tT3pDPWS\nyoy7ofmFt7yiryj1hZYo8CpdLvBaMFkJmlREM3hrl+UwW7YwV7ZEvyghCYNH0uua44r+t1KChvpw\nDpi8fVXbDeSORqFYtTFk6lio2h6Dp6GQ0YX5ylIGfW+0mNTOCjr14KWBWfDZcCkmoQUXzUJGTwGD\nx3/6jonxz6X7Zt7QYDkMf/vNRzBRyOA/XvcU8ZxKDxa1ug25DwtYPKfgpQI53igNCyky6P7arrmJ\nP953JtGkArNbEk2/t5j/3YrpYJqwRDf7zAatbsf24JHJihEsnPKmJgwJ4tg/Kq6oaCJ5jq6pgZiE\nTihpucDjhVvwNPNNVvwePFWpz8EDOBs27TF3VGQBEMYohrTdxZojCtjlBRM12xXMRsOgczcYk5DL\naCjJDF7VxlyZT/R0TYXiOXzKE9ti1f/sjK4K05VWmZXomAT+WFMXTcOLSYhj8Ji/4gtQIH1/Bgvb\nYWK/ghJN/ruQaErb56/2RcsV6XG/b8x/XJ7Yt7rKd8uWI9jq9Y+F4XqGRO1OcKmXVAYVPNPFmthW\nTaE+p+BEbaFKixC9KfCShOsGiqYokxXLxfKhDAB+jfLCKlqSGvfepAKQTWWSFHhxLpryY41QqjlY\nluf3KWLw6P6w1EHHIhtzPSUBY3whUlGUvks0aX8yum+ykjQHj1yc02KykiTmge6buYwO25Nph3tx\nl0IPHt2nc0a9LH8An9X18yb7vEESyh1KgEX0UYwzdlLQ53fLRdP17mtEsgwkmgN0DcIiP6bAG8rq\nWLcsh/NXDQX+v3okK36PY/+o+Do8w9k1UZRIvWyuG3SVbBVUuJVqDgoZrY7eph68oQY5ePQaXfMn\nDmFXTF1VPYmm34NHUq4Jb0JJfXi1CClcRkg0gzEJeUMPFIdFz2SFjh3gHS/JxUzu/5P3o9UA0MYx\nCQ0YPMuTaDaISQhLNA21nzEJrmBlohi8XCSD5/Un6n4Bd2i6hH/8/g7RfwNEM3jyBKfVVfr3fW8H\n/uuu/ZGPiRy8NldWqZdUhq7yPsqXfeyX+OSduwFAOHWGvy9ikhaq3Z3AhRcaGrr5xRjmECqWg/GC\nV+BVbdSc5Awe3ZNoYapmu9jv9b8lWa0VBV5EyH2zSS9jDMWaLbZ9rmzBNDQUzpACr2w5yBqqH8HR\ntosmNyHoN7MgL+K1khlG4ei04NdPcMWJ7JjYQKJJDF5GQ81hWKg6YmGRIBjsNDN4QqK5uFmfSwV0\nPKgf/0ySaFrSd99J76XoNe3S9SsWbpVBgTdAl5GV5HlRMDQVv37n8/GSy9cE/r96VCrwYl6by2gY\nL2Sw5+SC+BvghRQNGHaLJithaKoiGakYdY9HSTTVuh48/lMuyOar/uqk5TJkdBW6KjF4Eos2MWQC\ngHDSpIlnRno/U9ciYxKoUJvymMOSJ9GUtzejq6gzWTF9Bk/+fysIT6wVRRHFbmKJZhMXTSoYZTv+\nxYbtMCExDso1w/0Y9U5xVBzYjos7HjuJz/9qHw7PlOplhUwu8JJP7MOYr9ixhbrjrfTpDYqURqBe\nUhmUZ3l0toKD07yYoc8InwNiEaLbPXh1Es34fZMnxVEMSFkq8GY8MxgzwOCpsTl4sskKwL+LI570\nO8k+hye4NccVx7vZZL5iuWAMWJbn284ZPJW77J4BEs1i1UYho0NX259YCxfZFPQG0YJbRnbRTLBN\nNceFoaowDa3vBR6pZ5IwkDWpwLMdl0cfhc5Lf+Lb/oLE7Y+eEKZsvUBYlj/owQtCjjdKS94kodyh\nRNOR5LndkGiSaV6noIUrEte04kmRJgwKvBSiWUxCHFZJDF4jzfC6ZTkxSQqYrBCD16LJShQo8Ltg\n1ve4CImmmawHjyAzeLY3UTM0qQdP6oOb8CaUk/N8QilWdyWTFR507sB23YBEM0sFnidjLVa5ycqI\nxOAZIaZM7v+Ti8jWGTxiqXwW1zdZiXmNw2WmPAfPD7u/e88UPnLbTvE8MVB4h6CfQeeW44piXN4G\nEb6cqR/sqQiQGTwqcGbLlm8MEiHRlFewW5GTVCwHNceNlUC6rmcy0aZhDfWSyjA0RciR6ZzXYxi8\nnvXghdQ2KWsAACAASURBVI5lo4kyFc+aqtRNkC2HD7hjXpFERkmJXTQdPnDTfWrXyXnxWJJJRZjB\nsxxX3H+aMbn0nY97Ek3L4Vbu+TOEwStWbRS8nmKgPQaPepzbPf+7CdGDp6vewlhyiaahq6lg8FzG\nVRs0/jbafNmQqupZxBerYQaPTIba26+a7eINX7ofX73nYFuvTwKS4fsxCellG/sBuvWS6Ue/rzMZ\ncu9bEtMqgN8zaKHSksb0ziSaTuTv7YJ661XB4HX8ln3BoMBLIfz+q9bOqjUJGDwAWLvMfx5NnOQM\nPbtFk5UoUDEUjkgAgN+6ZBX+/LkbcdZ4XtrecA8eFXj+dsg9eJbD++bI6t91GcqWz6ItDzN4DUxW\neB+ULNGkAk9m8GyR70fvEzRZ8dlDmZ1otcATmnTB4vrFetzKHRU5Q1ldyPsA4LsPH8XHbt8l9p2J\nHrz+B53XHNfvuYjIwctHWLfbEawSGe/Mli1RADaXaCYfABaauFTankRTa5PBCLPHAC94qBCiAdTv\nvwhLNNvr9WyG+h68BiYrwghFq2OPafvzGV4YTZf4NSwv5uhaAxdNj5WhHrydJ7jyQFVaNFkhBs92\nBRPfbNJb8ibLxOABXDp8xhR4NQf5jCbu9e1IpHwXWX8i2i+IHjzveko6GbZsxiWautr3XjXXZV72\nKf+70aTZliJlqPeu2wzeQtWGy9D1nE0ZstEGMMjBC0NINBVi8Pq8QRLkRbakrQ//9KPH8OrPbQbA\nz2Huhh4f79TqdnTDaIV5ygRa/BpINAfoGvz+q9ZeJzN4jU7Idcv8wioXYPC4dXinJisAMOpFJYQj\nEgBg/Vge73jRRYIdUBRedAQYPO/MDEg0JZbCdnnfhO4xjxXbAWMQLBpJwqhIo5tP2GTFchjv5wsE\nnZNEM8jg1fXgScVHUer/kxm8dmMS/KBzX6IZN1mh4zKc1QNMLA3O5GYqGvhTEHRuO0wc51qEDDMX\n8Rj1u1HRzxk8PvGIYvCcQIHXmrkGgZixUkyPm+v6A0FbDJ7rCokcQVdVkbtI4J9Rv8q5eBLN+OfS\ndzScNeoYEJpgmoaGfEZvg8ELSjR3HucM3oYVQyi10YNXs10Mm4b3WOPXCwav4Bd4WUND3tS71szf\nT/CeZZ/Ba2d+5TJahOq/i2Z4EU9NuE2Wy1l0WvDrJ9yWJJreYlhGE4ufpZojFrZI2QG0z+DR/W++\n0sMCTzLaANrLYzyTIVw01fQFncv3waTM2Z6TC9gzyRfqaIG03RYHsR3dZvCYH38EDAq8AboIUeC1\n6M0qW37HGbQAYQaPgs4V7Dg2h4vf8yPsO1Vs+PokILYrHJEgwwjl/TVl8MohF01V4Vb/riukKXIf\n3GjOEDJLKnoyIQYP4DeHcNC5/JpSzcFs2RL5fny7VNSkCWIp5KJJaJnBC8UkyDeZuLmKKPBMPSAd\npUH50DQv8GjuIh/bfsYk0HEOsHQNYhL8wcArOhwmJiCzZcs3fIgwKJCLulYGADq2cYW6wzor8GyH\nBc5xAJEGRzQIhgf3nuXgtdGDN2Tq9QVezTfNyWe0mB685C6aB6dLGMnqWDFkJsvBkxg812WwXeYz\neE0KfWJDxvL+wk7W0JA3NI/5X9pSsoWqEyrwWt8f15MyKWnowZNy8AB+70xmssKQ0VIk0Uwgywck\nQ6qMHhgbShGGE+3GP1Dfe7fvLzLo/k/jwSDoPAjZGV1T+r+QIkMurJIyZ/MVWyxg8vEvaJbXDoIt\nGN2QaPLF9YFEc4CuI9sk6DwJGq04rB/Lid/lmIQTc1VULBeT89WOCzxiu8gpMQo0saVtbdqDV7GE\nZIVYN11TYNlMTMbyUpE7McSz8Oj58mfyz+PvzVhwUp2T3gPwe70CEk3d73WzHNdzhfRz8ABg7Wi2\nZXdDOcwb4ExmUonmcNYIxCRQ8XNophR4vYhJ6FPfDGN8oh3loimsv0U/htSDJ+QcxOC5wR48Nyir\nCPTgecV4RlNbmuzQBKdRD14nOWC2W9+DF3Xt0WfU9+D1RqJJ50orBV7B1OoYEBr0RYEnGLykLppU\nPPLnH5+rYPmQmVgmSZNc6qUEkLjAo2M7FmDwVLGItNRlmtQ3rEdcL0lBjFPSYqqXqIUYvKSW8pYt\nMXj9NlkJSTQbxyR4Es3QeFXy7gXtsCth+Axe7wo8K7SoNzBZCcI3WeEsXr+vMxnyeZVU1TBfsVGz\nXVRtR4zZcmtJOwgUmrXOr2GuzMFAojlA92GKmITWX0u9Z40KNJJomrrvNpZJMMlsBVTgDTVi8FS/\niAGCDB5dT/Lk13KYmJSRtbWuqbAkBk82dVleMCUGz4WqICDFlJk2WSaXl4pS+boeqZNo8m2hiR5N\n/MbzGaweyWLtslzLbnvCRZPC7mW5TsxgT0zdcFaHrvlGFzQoH/KcGKk4FkHnXv/iYsOSpEVAWKJJ\nq9JqxGP1Bdx8Jb7Ai5JojuSMluRK9P7xLpq+i2A7xXLNDmYwAtHmSiRNDThW2q44PrR9DxyYweX/\n+8cdu96JfscEJit0HRRMHTVP5k2gCUDWUD0Gj3rwQi6aCRm86WIN44UMchmt6YSCMSZeX5GOFTn7\nVptMeilQ/WypV5gKVaC+3ymN2HViHs/44O04OVepe4z6hqOul6SQGex+zztrIRl+Ukt5y4sqMfXW\nFn96gXDvT7IcvGCBR87N8uS73f2ihaP5XjJ4LUSy/CaiLug8TQxeG71vC5IxGM3jOl1sDkg0u9BH\n6+fb8r8HLpoDdA2CvWlj1WDlsOm9Nv45JNHMSwNDfRZX70xWCGT93pjB4z8pKJlkmtxeXuUSTUdi\n8KTibGIoI+Xg1TMlNHk1NAVXbxgX/5cHzHHJYGEkG5ZoUoHHP5sYvL94wfm4+Y1PR8HUO3DR9M8B\nRUg0G/fgDWV1L9suJNGciZZo9qsHzxLSoiiJZojBk100vUgBwTg4vovmXNkWhg+REk2LCjy9NZMV\n79jyxYX618mf2VYOntdLKiNc8AGczR8rZITEEQgWGLTAsfXwacxXbREG3i5oX6IMa8KgyeNwlkvF\n5HNKZvAKpi7+TtyDZ5ORRPC6TsLg0XaR9I7OAbonNZv03rV7CuvHctiwws8bzQYKvPQzeNuPzuHo\nbAW7vZ4XGQtVG0OmJha32pFru4x6qPs/MZcd+QB+HSXZJhobetWD98W79uN/fWdboudS/7uIxmko\n0QwulBFozKH7VSGjtc/gicl473rwZKt8YBB0Hoa8MMuVIn3eIAnt9L7RvGS+YvsMnqYIN9V20A6T\n2AgO86XnwECiOUAXIWIS2ggbX+EVeI2ulfFCxltRDxYsMjo3WaGYhPgCjz43ugeP/6TJ7voxvopO\nzeSW0G5zap9WLWUGb2Iog1PE4NmsjqW8ZtNyvPLKtfjum5+JZ2xcLv4vD5jEiMr7BARz8ML9f6M5\nA2eN5zFk6i1L5/wcPD8mQZMGe8txhWkKYV5INHUYut9XR/+nLDUqeMSxVdW+9ODRZ9K5EZRoxvfg\n8cFAhSbldvkFnhVwtJQ/B/AnOyNZo6WYBNlcIGxBDvg9SG1LNCMca6MWV1QVmCiYOLXgF3jyuUXf\n9fE5fr5PF7vL4DVaCKBJMUluaxH9jqZUGNHfBN6DF+OiGZJVAjzjMkkWnV/U8+t2QbpOwtsZhusy\n3L13Cs/YOBEoRnMZTdw344x30gRyP5yLcEEs1WzkTR2a1j6D57p+IHBSm/Reoc5kJWG/kk09eLra\nErtftR384/d3CNlxHH72+En8ePvxRO8pm9YAjRdWSJUiK1EAf+GB7nPL8pn2e/AWRaIZXNTr90JB\n2uDHG6XQZKVFaSRjLOBMbTl8ob5RH3bL29GFHjzRejEwWRmg28hK8rxWQQweFTZRUBQF65blAkwV\nsQj0+q5JNJsUeLrmZ/5EMXg0waRIBb85lzMfhrfyQ30HctG6fiyPmZKF2ZIlso5knDWex0dueCIu\nXjMS+L/cg7d8WGLwcsGYBN+EJcjgEfIZLbIoaAQ/28iXGdFX4bgMX7//EF7wbz8PrFIJiaZpQFf5\nKrQrFT+HQxJNOt66Z1Cz2KBJey5ixZYKCfoe5VW9OgbPdQMmKy7Jm7R4k5XRnNGai2ZVZsnqJzkO\n8xm8dgao0+Va3QQt6trTVRXLhzJCcgz4E7mMpoptO+FJ8eRCsB2EYxIaTSqEyUq2vmCvBHrw/GtT\nXszR1Gizn0ePzQkGXl4omiCJZpOBnIp6cvSlIidJDt6OY3OYLVt4xsbl0FVFSLVNXRXX+VKQaJ72\nJLH0k1C1HVgOw5Cpd6UHL2kx1UuE+6w11TetavY6Q+cSzVYYvG1H5vD5X+3DnTtPNnzedLGG0yUr\nUQHMxPHkfzeUaHr9u+HFWeoXpuuD3/Pam/T2ysRJhpznB6AjJudMRJolmq3mz1VtV8yb5ioW76vX\nFCHTb3eRqNKFflMZDkNAKt3pfLhfGBR4KQRNfto5qc6Z4IVQM5p608ohrJDYKZLpvOiy1W1/tgxy\nnGxW4GU0VQxm8kq534PHfzl7nBvDBCSamiqacwWDJ00iL1w1DADYeXJeaL2TIBfD4AVz8Px+KMHg\nhQxl2pNoBhm8oFyH4cBUCWXLwcl5v6dmvmJDVxVkDb4KbTsMJYvHRozlDUwVayhW7cBAAaBj56p2\nIcwBIsLMffkmPxcCGXmSnAPwcvAkkxWbGD5hMe5/ZqAHr4XJjrxyHWW0QoGo7QTQPnBgBtuOzOEF\nF60K/D88YQM8Bm8og+liTRRbNOlaOWKK8+z4LD8vppuwCs0QNllpyOCRRNOMYvD8jKsAg1eXg8ff\nf8fROdy3fxqOy/C7n7oLn7hzN4Dg4slEIYO8oYmIkzjQZ9Nikx8n0jwm4e49UwCAp2+c8CJc+HHI\nZTRxf1gKEk1i8MI5ZiVxz9LadtFkjHHGSaWg8y5scAcIR+EkdRy0SKLZoosmMXdTTRZTphZqqElj\nVCM4bvKYBJ7fp9aNa/Td0kR3WZ6rFtqZPNM9plRzepaZarn+PQIAnH6fSCmDcNHswK25V2i1B08e\nT+crNqy6Rds2C7wuu2jyhSt/HrpECTw0nn0P0BdQHkw7fXBvePYGuAz4/avPbvi8D/7OEwI3irPG\nc7hi/SguWzcKoPMCb7zAC6NRyWI8CrLRh6oq3LzEdf0wbm+wPjvE4FmOC8Ob7Ad68CSJ5gWreYH3\n+PH5yB68OJi66jXohwq8UEyCb7JC7GGQwRsydRRrNhhjYn+awQkNduSeCPAbPU0mTi3UcM5EAQDv\nExvO6lAURQTWE6t38ZoR3LVnCodnykK2SwWQHsOc9Brh3hErJMMEJImmG3xM1/zjUao5YkI2W7aE\n8xX55cgTVjLUGMnqrZmsNGPwvAGKAmhb+a4/8bPdGMsbdddq1LWnKQomCiZsl2GuYmFZPiO2Z+Ww\nie1H5wD4DN5UAwY/CeqMDxpMDikupBDR2yZ68DJagIUL9uD5vVL/90eP4dBMCTe+/upAASW/dnzI\nFNtXqjkYzUVf1z6DRwUevyZGEjB4DxyYwbkTeZEtmtFUVCwXWd3fj6VQ4J0u17yfwQKvKFQHeuSC\nSBLI9u2q0jiUezFA9xHfgThZX2zNk4m16qI57fXDTjVZTKHFlpliremCZ0sxCS6pWKIZvPACR81x\nAwsrSRBY4Ko6GM13nxNwwn3XKSpg0gB/YRapY/DKloNhU8d81U5U4MlM8ELFhkMSTSm71mjtFBXb\nMZozMFu2utKDJ/JtBxLNAboNU5rct/xaXcNfvuD8QCZeFMYLGdGvBwBvf+GF+OafPQNneb1unRZ4\nV6wfxb+/+ol41qblDZ9n6EpgQmwaamC/hUSTevCIwXP4ZJ8KQuGiKbFoa0ezGDZ17DwxL7KOkkBR\nFMHGBQu8YA+ecDCM6P/jf+twWWuacDtU4KhqUKJJvVWyBHe+YgnZme7FJNDAfIknPz00XfJjErzD\noDVwL+wlhEQzIibBL/7oMcmAxev3IbaZJq2aqggGj+v544POR3OGkLAmgTzBiYq8cEIDQdLDeWKu\ngjseO4k/fMa5dX2qcS6aE57REMkvqcBbNZL1pC8ujlOB1ymD14LJCn1HURJNGmxzhhaQPgddNH0m\n+cRcBYemSzgwVRKPq0pINl3IiPOj0WAuf+eAzOB5BZ5U6M+WLJyWDGwOTJewUTJXoZ7BXMbfj7jo\njDRhLobB812HdRGg3CqD5/cGpSToPMzgJdwmy3GR0RWYen3MRyMQgzfdgMEr1xxx/5cNkuLgUkxC\nk2gc2m5dU+tMmahdQWbw+N+tM2PyhHyuR0YrwlhsUOBFwpFaK9LgViujbLkiRiYJcyb3tM9XLNhC\nosnHvHYNdsqWI/JKy22c52EQkz6QaA7QdYig80U8qRSFRw6c7Uk8O8ngo/d7+RVrIx0BZcgmKwAv\nUOXdJvkJ9eDNSwwe5eARg6co4agFBResHsbjx+dF1lFS0HdA7p2qAgxJxaMck1COcPAE/OyuVvrw\nwgHTslzHZUxM3KdCZhvDpuFtF38uTT4u8gq8o7NlyWTFdyjtRw8efWakRJNiEiL68xzXDejiaUK+\neiSLuYoF23F5bqCIlfA/sxrqE0s6kVuoWGJhIIrB46YIvp1y0oL5sJdNeMX6ZXWPUYEqX4KqqojF\nBmLn6Lwilun4bEWwSs1kY81Ah6cVkxViJ+RjS5bVWUMLLIDIJiuy7GhyvgrLYbh//7R43NDUQP/s\nuOeiCTTug6tYQQaPJqhZXYOhKQGJ5l99fQve/NWHAHAm6tB0Sdxz5OOQNVRRkHdjpbjXoN672VI0\ng0eKh3YcdV1p4qmmIAeP7iN0riSVTQuJps4XvJJKEYmZm2pgaCQ/NlNqXiDJuYL0d/x2U0A7fy4V\ncuGYhNEcH8Pa6cNbCCxw9WZBwwr1nfdKCrpUQdcVMeVpkmhWalJhleB+uBCSaNok0ezA6Ang5/po\nPnmh2Qw0rhP5sEQJvEGBl0ZctHoYL7l8NZ4QMfnrNVaPZEUuyWIgo6mBz8oaaoDRowJxlbddvoum\nJ9FUVRFEXsjodfK4C1YNY+cJT6KpJ98nmkAu91jO4awRyEKRTVai2EPAl5W10odHkyxDVYVUhz7X\nDUg0/YnDnCfRBPzjRavF507kkdFVHImQaMrmFjPFGo7NBt05ewXL9g08SFJKaBSTYDsUikoFHj8X\n1i3LgTHOUuiqGi3RtB2Yuirkz8ktnW2sGuXnQNQEx3EpELW1DKfjs/z7Wz2arXuMBju5R1ZTpALP\nm1jSJH3lCP//Xi8aQVU678GjY0fbkiQmIR/lolnzYxFymWiJJpn9WI4r9u0urweOXAJl9n2iYCbq\ngwszeCTvNrz3kyWau08u4MGDM3BchpmShYWqHSjwSFUhxyQsBQavWQ8e3bPa6e2hy8s3f+hwYzuE\nH3TuyarUZHI223NkpkW1pIs/dI01MjSSr8NmbpuAL9FMFpPgG1QAPNJHVxWx6BFe4GhFmk4IOPX2\nyEnTDsUkDBi8IOhw0FwgbRLNZV5hlUSpNBdaMOBKLFVi8Nov8PKGBlNXu9aDJy8mDySaA3QNw1kD\nn3zNVQEJ5WJBUxX8ybM24NqLVzV/chdgaGpgdcTUg4zedZeuxpuetxEZXcVw1qgzWaHBfK5s1/XA\nAcCFq4YwU7Jw9HS5JQZPFHheL6Hcf0fbTZIgGlDDgbM04W1l5ZMmWXRz0dSgoxqtCE8FJJp+gUf7\nSBPlkZyBtaNZHD7tM3hyiDwNpu/69la86tN3L8rqoDwRk3sZASkmgUxWGvTgiQJvjBvwTJcsHmYv\nii3/M6uWC1NXxUQ9qZPmQtXGao8hK8W4aHLpjP93EpCUkt5bBu3fGqn4C0o0q2LbAGDVMH/e7pM8\n6+y85YWGrEISUIC1yEhrYrKSkSbIARdN2xWLNmSUwr8j/xqn4kJmHR84OIOVwyYuXD0CU/evc0Xh\nxkF0fTaaVIhoDCrwvHtHRlNhGpooRBljgv3cP1UUsSJnRzJ4mujRXRIMXkyBR+cOsaq0UNYK6FzX\n1OSh4r0EfZ/0XSVl8Kg/m87P8LGKAy2iNVpMkc/pJBJNh3GJZrKYBM5+EGNZMPWAc7Mfk9DcVCgO\n81VbvH6h2huJJk3qyXAt3Bd+5HQZb7/54a5M3AG/T3mpQPTgeeqUfl9nMsqWgyFTR0ZXW+7Bm6vY\nsIUqp7OQ+4rlCgOsbsQkyOZpQOeKtn5hUOANUIe/fdFFuPaSxSnw9BBbmDWCEs0rz1qGv7nuIgDc\nHGFexCT4OXgAMFuuRWbukdHKY8fnWyrwqFgrmLznRnbQBHjvYFXqwZMnuIShdhg8b3DTVQWGZ89O\nq0cLVVsM2sE8NEs4AxohieZwVse6sRyOni6LyUIg6Nzbh4cOnsbhmTLu2nMKAPCfv96H1/3nvYm3\nuxXQZ2a8Aj3cZwfEMHieSyaxSmQgsW4ZL/AOTBUxZOqR/StV24VpaILBS7qaPV+xhQQyygWPcsCS\nuN7JOD5bRkZXxeRJBhU/q6TiT1UUjOUzUBT/uy9VHagKROG3xwuzvnTtaMBtsx04ricJStALVLPd\nAMsWMFmpOeK7pAUPU9eCLL0nD5SdYWu2i3VjObz08tW4eM0IFIVHoizLGdA1NVHYeJyLZkZXYeqq\nmPBOF2ti0WH70TkciijwRA+eoYke3bSbrDDGRLFC1wqhJJmsAG0yeJLkOw3ufuSUrEj3tyRknOW4\nyGiKUMzcv38m0ef5cvlGEs3WGDzGmJDiAc178AxNheE9uWByA6A4Bq+tHryKJRahesXg0Xmja7xN\nxAq1Ddzx6Al844HD2HZktuPP+vnOSVz9gdux88R8x++1WKBzgMaZfl9nMso1B1mvvzrJmEo9eMvy\nBu/BcxgMzZdodtKDR33evXDRXKIteIMCb4D+wtCCpip8dTz6ahorZETvkuU5iNFkeLZsRTJ4Tzp7\nTDChSWMSAAQmpQWzvsCjHjzGeAZf3qz/bFodb0XK5biu12SvCPMOOh6T8/5E4lQMg0eMC/V7DGcN\nrFuWw5GZMg5Nl7Esb4jjQDEJpxaqglH6xgOHAQC3bDmKOx+f7AlLQQWd7mU4BSWa5IQX7bCpS6zS\nbIjBO12y8OwLVkiugHKB53jMjSr+ToL5ioWJApc+RTGxIlzdOw8fPTaXqKA/PlfFmtFspONmVIFH\nnzGe97PwFqo2CqYuvvs9HoN3ydoRuKzeObEVUL9jlGFNGDXH4QUeSdxCLppZcS3xn6YRHHZ0jUd7\nyOc3wHMs3/z88/Hl118NgN8rJjyZas6gPrj4Yx3noukXeHw7j836heX2o7OCwTvLi2YBAFNi8Ghf\n0p6DV7Fc32U23IMnGDy6b7TejysvGKmqgn4TC1TwEHiB13yfLJv3c1+6dgRDpo7Ne6cSfd6MkEo7\nsZNKMsXK6Kpw3WyElmISvP2lRc6hMINnO9BVRXK3baMHr2oLJUHPJJrePV4EXocYvMMzvHXg0Eyp\n7rVJsXnvFOYrFm595CgAvuC7VCAKPJUvuPWhbT4WFctBLqMiZ2gt9eCtHslyiSYt2qqtLZCGUa45\nMA0VWUPrismKK+XbAkjsjJ02DAq8AfoKOQcP4BOouGvpBRetxIMHT+PAVBGM8QHBEAyeVdcDR+/3\npuduBADcl3BlFvAno7kMD2gezYUYPE0FY/yGVKw5kZ9NDF6U+2IcqOkY4BNfOfSWJsA5QxMFHmMM\nCxVbfJbM4CkKkDc0rF2Ww8n5Kh48OIPL1436ERQec0IW+xeuGsaPth3HyfkKth/lq6X7vL6uboJW\naEmiKbN0wiiBHFIDDB4VHcTg+T14hOsuXS1uyrYbZvBUUTgmWc1mjPd2DmeNyExDeYJL23TDZzfj\nU3fuafreJ2YrgQJOhuad0yM5PSBrBDhbNyW5aBYyupjA7ZlcwGjOwFrveEy3INO0HDdgbuC4CMiD\nG8n3LJvMHkii6T+3YvkMHi14mCGmm4qLk975ff5K7l65fiwXeJ6hqRj3HNtEH1yDayvM4FH/bkZT\nMZIzxHGk7MCMrmKHx+AtH8oETJP8HjxVfH6vGLwDU0W87esPd1xAEnu3ctjEXMUOTJ7Czr/tMXj8\nJ7dvTy5P7hVqISMtVVWQpKXHktQgTz53DPfsm27+InDml87tONfaqWINhqZg/bJcQpMViAU+AA2L\nZttl3j2UGDx+LxBB5zVXSIqB9nrwilUHq0f5ddhrkxXKQwvfa6jAOzjVeo84Ywwf+snjuOGzm/HO\nb27FHY/xUHpi6ZcC6LpUFHLRTBGDR8xZQmnkfNVG1uD38XlPosnjrjwjs5rdFotXtfl2ZBMWms0g\nJJoDF80BBmgfhqbUM3gxF9Mrn7gOAHDz/Zxl0iVqf7ZsRbJoAPBqL2fsyhZMa8gQIp/R8M4XX4Q3\nPPu8wOPEVrz1a1vw4+3H6yISAH91fPvRWXzhV/tgOS4+cttO3Hz/odjPJdt9gPdgrRoxxfGgAu/8\nVUNiQlGxXNgukySafLumSzVPrqiIAmjXyQVcunZUfJbuxSSQ9OXvX3YxqraLD9z6qBh0955aSHbA\nWoBsZ27oIZMVr4jjrq5KqOjgxS8dH+p9IQZv3bIcLl3L5XyqEuxf4T14mpigJ1nNLtUcuIzLXIdM\nvW6CQxNaysEj3JtggnhsrhzZfwdASK6GTF5YyqYLEwVT9NeVag4KpiaK+1MLNawfy2F5wY9TcF2G\nX+6abCjXrNkubvjsZrz+i/f7++a6nkSl/ljWvd5xYxm8iuVI8kZfoimDiouTc3y/nnT2GID6Ai+j\nq8LVVkg0G/XgkUQtXy/RvHj1CLYfnQVjDMc89vqajRPYdoQzeLLBCuD3ddGEPpfRY4vLbUdmE7NA\nUfjy3QfwzQcP43sPH237PQBflnm2cCD2C4xSlbsO0/60k4kZ6BfuYW9QzXZxy5YjTSXHNYcFZPJa\nk/MW4AWA5brCifLq8yaw++RCQCERBctxMVexsclbjIiTaU4v1DBeyGCskElmsuLd/5NKNEkFAfg9\npM22MgAAIABJREFUeCWJwct6rAaQvO9Y3paFqo0VwyY0VQmcP92ELRk66VIuLoFUOwfbKMp+suME\nPnbHbqwfy+HWrceEvP3g1NIp8AIMntL/OBICY4wXeBndY86SBZ0PmQaGs7qQaMqLtjd8djPec8v2\nlrfFl4qqbTHVYbjUojCQaA4wQPsYzRlCYgZQD1701bR+LI+rzxvH170CydAUGKrP4EVJNAE+oXzg\n76/Ff7zuyYm3K29ogmF6yeVrcNU544HHaVD9/iPH8OzzV+AdXp+gDGL1PvuLvXjf93fg2g/9HB+5\nbRc+/NOdsBwXb7/5YWw5dDrwGspyA4BvvPEZ+IsXnC+OB006Llg1jNMlKxBoPhQyWZku1jDsTfxl\nhuuydSPid7qpPnL4NM4az+GZm5bjglVD+M4Wf2K5d5IzeDtPzOMLv9rXlTBjn6VTYahqwLVOZjCN\nkNMhySFXj2ZRyGjYd6oIXVWwZjQLVQFedNnqYP9NoAePu2hScZFkskMF3VCWy3TDDJ7f/K4EmrAf\nOXK64SokYwwnPIlmFKiAHcrqGMrqgfeWGbyFKmduyURkopDB/375pRj3iqDpYg3fe+Qo/uA/7sWP\nth+P3Z4P/OBRPHBgBnfvmRJSM4cxsaqqq2rkpGK2bOHFH/0l7njsZLDAc/wBtmK5yBk0AeXHPhuW\naHqr9pMLFYzlDWxcWQDAr3cZv33FWlx36WoAfo9sY4lmPIN32fpRzJQsHDldxvHZMnRVwfMvXoWZ\nkoX7D8wE+u+AoIsmABQyGspW/WczxvCXNz2E///LD7TVC8IYw4938O/qa/fFLwQlAckyKfrmtMQg\nLVSdgOuwprU+caR7geLZiTPWm7Dz7z9yFG+5aQt+tftUw+fxXrqwRLPx9jguA2P+ffNpG/h9/p69\n09h3qog3feXBSMk1GaYQ2xzH4E0XaxgvmBjLZ9qKSWj0nVgOgxx0PmTqKGR0KejcCTB4rZ6P9D4j\ntMDVMxfNoKtyeKFBlmh+5Z4D+Iv/fihxf/F9+6Zh6iq+86ZrMOwtlm1aOdRWsdgvUL1L7Rr9jiMh\nVG0XjFHGaTL3yvmKhZGsjuGsgQUvJoG+d/64jXtaXBwThWY3GTzGvAzigURzgAHaxrtfegk+esMT\nxd/kUBeHP332BjG4jhdMEX1Qsdy6HDoZE0OmYLmSYN1YLlAYhUETy+dduAKfeM2TIk1paELLGLBx\nRQEHpkq4YNUQjs5W8J+/3odvPHAY7/jGw3V9ZjTBz2U0r0eRP0YM3oWruHHMdLGG+ao/CAO+rf10\nsSb2d53EhFwmMXia99yHD83i0jVcuvmqq84CwLd37WgWez3jjvd+dzve9/0duO3Rk02OXHPYDSSa\nZJ5D27BVaqwnBs/QVDx943IAvAjKZ3R8+fVX4y3Xni+eG25Gr9qei2YLkx1RPJt6pKmGvLJK8sLh\nrI6K5eLRY3Ox7ztTslCz3ViJJn2Hw6YeMI0BgOVDJo7PVXDTvQex49gcCqaOkayBz/zBVbj1L5+F\np543jomCn5dHRUIcG/TosTn81137cdm6EdQcVyw4OK4/uKkqPxbhVe+fPXYSjx6bw2zZEoY5QH0P\nHhVjssmKDE3lcufjs1WsHM7iqnPGMWTquMgzSCL8r5ddgldcuS7wXqWaA9dluOneg3Wyq0YmK5et\n5Qsd247M4Zgnl33VVevxoktXo2a7OGeiEHgvKhzo/MlJvU4yNu+dxt7JImbLFn60Lb6ojsOOY3M4\nNF3GxWtG8ODB09jVgRkESZjPGef7IrtDlmpB1+GkjpMyHOn8F66PPZh73udlIt63vzEzziWa/uCR\nhO0IZ+ddtm4U+YyGe/ZN4dsPHcGtW4/hFzsn6143U+THctMqYvDiJZoThQzG8kbimARVTR6TYGi+\n6VQ+oyFv+vepqsUlmu0yeGKBy+vz7V0PnmQspgXdXEs1WxTPh6ZL+O97D+J7Dx/FzQ8kW/zYdnQW\nF68ZwfIhE+95+SV403M34rK1Ix318zXDzhPzXW1tcMRCClIVdE6FVI5635L04FVtvnBp8vOJ5ju6\ndN3umyqKRZVi1W4qDbYcBpfxe3IuIZPYDGKhZSDRHGCA9rFi2AzIoUZyhhiQovCCi1dh63uvw+1v\new7+xxPXYeOKIdGzV4hh8NrBG5+zEbe8+ZkNt+Mll6/GR6TiNAxd8wuK//PKy3D/31+LL7zuKQCA\nD/90FwxNwc4TC3j/rY/iwYO8P9B23YCFPOBPtCcXqsgaqjB/mJyvikGXWFA6dpPzVfE/ylobNvUA\nM0Hs5/G5Ci5fzwu/Vz5xHXRVwVPPG8eGFUPYe6qIbUdmcdeeKeiqgvffuiORBKLRSn6tkUTTy3YC\ngGduWo5HDp8WTITjNWQDwHMu8Ao8j6W8ZtPygBHOWD4TKLKEi2aTyQ5jDO+/dQfufPykyOwZyRrR\nEk0pgHbjCj7R+/D/dyUA4IEDwX7P2bKFd397K07MVUTPV1QGHuAb5Qx5BZ7M4F2+bhSlmoN3fmsr\ncoaG1z+TS4evu3S1eL+xvAFF4fKku/ZMYTir447HTuL+/dP42n0HA9/NjZsPwNRVfPzVTwLAV7z5\nvvnnoaYouHHzQbzkY78MsBk/ffSE+H3HsTmJwWMoVm3cuPkA5iuWcC4VJivhHjzv+z42W8aKYRNX\nnTOGbf9wXWwBDPABN6OrKNcc/GDbMbzzW1tx/afvElERAGdtDU0RMsSARHPNCDRVwfajszg+W8Hq\n0SyyhoZPvuZJ+NRrnoQ/vubcwOeRvJcm3oWMHpjQfOehI3jS//kp/uF72zGS5dfZF369Dx/4waN1\nLP0nfrYbT/7H2/Clu/fXFVU/3n4CqgJ89IYrkdFV3PDZzfj6fYfAGMM7v/kI3vXtrUKy1gxU0J3j\nMXhygVesOQHX4faCzvlP6sEDehPCTJLnZtLnKJMVme34yj0H8JabHhLn/033HsT1n74LgM/gGZqK\nq84Zw+a9U0Jm+4tdp2A7buD6J3n4phX1Es13fXsr/ubmh8XzxgsZjBcymC7V8KGfPI5f7YpnIsm9\nL0lMQs3hao+MJt8vfKXB6TLvEWxlUevUQhVPef9tuPWRY4KxK3j3ofke9eDReEILBbIs/4jH3m1Y\nUcDxuQq2H52Dpir4px8+VmccFAZjvL/8Um8x51VPPgt//cILcfZ4HkdPl9t2bGz2mW/40v14600P\nde09RdC5mq6gcyqkWimsyBBuJKtjoWajZruek7p/3TLmm+D8yRfvx+v/675E22HqKrJdiknwpdLe\nIufSrO96W+ApivIiRVEeVxRlt6Io74x43FQU5Wve4/coinJuL7dngPTjzc/bhE+/9qqGz8kaGjau\nGIKmcmvrv3/pJQB8mWI3kNHVOmMVGRevGcEnX3NVw+cAfNBdPpTB1edNYPmQifVjeWxcUUDZcnD9\nVevxwktW4b/u2o/f+eRd+NjtuwIMHoH+npyvYqJgisDrj9y2Cx+/Y5f3OXw7nnLuGIazOqq2K46H\nqWtYOWzikrUjgV4x+XP+h9ffuGLYxI1/cjX+6toLsGFFAXsni/jUnXswZOr4yA1XYv9UCX/99YcD\nLE0Y77llG17+8V/FTiiCkpygRNOSJKrPPH8FXAbc7U20iMEDgGedv0Ic3yi87ppz8ctdp0ShVbW4\nRHPM68d66KBfgM2WLbz95ofx7m9vxV17pvC5X+7Dn3/lQXzwB49CUxWcM5FHwdSwULHxzQcO4zWf\n34yP3rbLD3pWFbzk8tXY/g/X4dpLVmHNaBa/2nUq0Mfz8Tt24Sv3HMTnfrEXx+f4pCW+wPMlmgWv\nj5Lwu1etx9b3vhA/+atn4463PQcviMir1DUVb3jWBvxy1ykoCvCPr7wMVdvF9Z++G3/7za341588\nDoCzct9+6AhefsVanLu8gItWD+Pe/VTg+ecHGXIsVG3BStVsFz9/fBK/4503AGBqmnjsk3fuxt9/\nZxt2nlhAlgyLjGgXTfqc47MVrGwh+zOf0TBXsfFvP9mJ85YX4LjAG298QEyIOGvLYw0yuurbsasK\nsoaG81cOYeuRWRzzCjyAf5cvvnyNCO8lLB/OYOVwNvDZJGEr1Wy8/wePolSz8djxeVx/1Vm44aln\n4ZHDs/jsL/biAz94VLxu14l5fOS2nXAZw3tu2Y633PSQmGgWqzb++96DePrGCVywahhf+9On4ZyJ\nPN79na24/dGTuOm+Q/jqPQfx2x//dSLDizqJplzgVe1A37CuqnXuhc0QdtEEup+FN7VQxZ7JInKG\nhi2HTje871heLyhBlmlXbQcf+slO3LLlKLYemcXBqRLe+73twmBKZv6etmECO08siHvEL3ZO4s++\n8iB+60M/F8edCryzxvMwdVX8fbpUw833H8K3HjqCyfmqKPCW5TOo2S4+dsdu/MuPH4vdB2IOkvTg\n2Y6LjO6zHwVJaTBdrOHefdN42oZx32Ql5thNzldF0Xvz/YcxOV/Fv9+xSyxwDWW5SqBXPXi/3HVK\n9E7rmgLL9R11SZ75jI0TngQY+LsXX4SZkoVvPni44fsemi5jvmIH+s4BYP14Hi4Djp5u3bSlGfZM\nLuDAVAkPH57FyS7l7VE9RwVHr3pd79p9Cm+56SGx2NwMVEhlDQ3ZTLJ4AjKEG84aYIyPvRQJJWPH\nUX787t47hXv2TTdkRCtSoZnVk8U1NINDcSUqFXhLs8Lr3ow4BEVRNACfAPBbAA4DuE9RlO8yxnZI\nT3s9gBnG2CZFUW4A8H8B/F6vtmmA9GPlSBYrG6zcR+H1zzwPq0eyeOLZyU1UFgtPPHsMl64dCRRT\nz7lgJfZM7sMrrlyHp547jsMzZXzk9p340E93ciOJQnBySfeW+YqNcycKwir+NolBIbYun9HxqqvO\nwhd+vS8gSX3Hiy7CqpHg5Jlc+p51/nLhvAjwCQ4AbFwxhIWqjVu3HsNfPn8TXvaEtTh2uoL3/+BR\n7Dm5gDc8awNedsWagOTuZ4+dxJfuPgAA+PTP9+Ct114gPuu7W47isnWjYkJLTplBiabPHF151jLk\nMxp+tXsSuYyGIzNlXOjJ9s5dXsDZ4/m6+ArCHzztHHz2F3vxwR88ik+99irMlS2YuoqVI1lcf9V6\nfPGuA3jt084BALzm8/eIicQvdk1ivJCBqnDX1X++/gnYsGIIBVPHrpMLeNvND2M4q+PuPVO46hxu\nBqJ5ZiTEiDz1vHHcsuUonvaB2/Gp116FC1cN44t3HYChKfj6/YcEMxVnsrJ82ISicJMd6huRMZw1\nmsqN3/WSi/GE9aOYKdbw8iesxUdv34UhU8emlUP4xM/24EfbjsNxGUo1RxyHp5w7jm8+eBjfeOAw\nHjgwHZDNANys41sPHcYrn7gO33noCBaqNl5y+Rr8ybM2QFV946FHDp/GbTv8c5MYPFVVPEYhyLTT\n9z1VrGHFSAsFnqHh+w8fxXzVxuf+55NRrNp469e24PbHTuLUQhU7js6Jfr+sroq8PmLhLl07ijsf\nP4lizcYLLlrZ8LPe9LxN+J9PP9f/bNOXJH3m53sxOV/FN974dNRsF088ewyKAqwczmLXyXl85ud7\nsfvkAlaNmHj7Nx5BwdTx0796Nm5+4DD+6YePQVcVfPj3rsRnfr4Hk/NVfOYP+ALXE88ew4d/70o8\n91/vxFu/tgVZQ8WnXnsV/ug/78PX7jsk2Ns4zJYtqAqw3ru2Z0MFnixpb4/Bkwo8pb7AIwbl3OWF\nwEJMsWpj54l5nDWeF4tVcbjfW6D5/avPxn/8ah+2HZ3Fk84ew5HTZUwUMtg/VcT+UyU854IVqEny\nbtonKkK/9/AxTHnOwl+86wAOTZdgqCoq8NUEBOrDsxyG5164Anc+PokjXjHwyZ/txjtedJGIPJgo\nZLB8yBQS7Vu3HhOyz/ffugMLVRvrx3KBPvOHD89i98l5bFoZlCADnkpB8SWajdga6tdeMWTC1FVs\nWF7AsdNlFGs2btlyBJbD8LtXrReqhagJ+Jfu3o/33LIdF60exluvPR833XcQ+YyGx47P46feNTxs\n8l5gOacyKWzPCCYOeycX8MjhWbz7JRcD4PeCWx85hh9vO47vvOkawVY/Y+Ny3Lj5IDK6itc+7Rzc\nsuUovvngYfzxM8/D7pMLeNe3tuJvXnQhnnKu3yu/zXOClvvOAd906OB0CSNZA5/42W780TPPa9iS\nkRTk0km/3/DUszt+T0dcZyTRTHad7jg6B5cxXLaOF7gVy0HNcSPHzKrNVSEHp0u4ZctR/M11F+JN\nz9vU8P19iSblzyXrax/OGmLxeaFqQ9fUwBi3LG9gx7G5QN/btx88jL9+4YWR7ykKPENDLhMMXN96\neBY33XcQ73rJxZE5yVE4OFXCY8fm8fQNE0I9Myjw6vFUALsZY3sBQFGUmwC8AoBc4L0CwHu9378B\n4OOKoiisF53aA5zReOkT1vR7EyLx+T+sN3b5o2vOxcRQBk89dxyqquDsiTz+5forcMGqYRyYKonC\ngSBPxJYPZXDWWA6vufpsPHPTcmw5dBr/ddf+QLHwB08/xyvw/Nddf9X6uu0Y9ViKt7zg/LrHAAiH\nuBdesgpv8Qq1Nzx7A9aN5fDhn+7E225+GB/84WM4f+UQ5qu8R6tYc3D+yiGcv2oIn/zZHty/fwab\nVg7h/gPT2HaEr5aTlFb3wuG3HpnFB3/4KPKGjq1H5kRhkdFVPG3DBG7cfBA3bj6IDSsK+PPn+oPO\nv77qCsTNHQqmjr978UX4228+gqd+4DYwBrzxPD74v+O6C/HDrcfw+5+7B5bjwnJc3PzGp+PvvrUV\nu08u4C+evwkvv2ItDk6VRG8lTVCfdf5yfPzVT8Lz/+1O/PEXuXSEbMQJ7/vty/CyJ6zFR2/fibd9\nfQtG8wY0VcG/vOoKvPmrD+EDP3wUa0azsWzVk88Zw+a/ewFWjWRx7vIC1hxvb+LxsiesFb/f+hfP\ngqmrYOAulT/efhyaquBNz9uEKzx57vMvWokvbz6At9/8MFaPZPGul/JJ16df+yRsWDGEH2w9ho/e\nvgtXf+B2nFqoYt2yHK7ZtFz02AHAK65ci1u2HIWiAG963kZ84md7kMv4X1LB1OokmvLgTpK3JMhl\nNBydreDlV6zFtRevhO0y/POPHsNf/vdDYpCnSdtLn7AGdz4+iQ0r/N66F166SrAAcWwqIZ/RA9dh\nPsMla5//5V589PZd/6+9O4+OqzzvOP59ZtO+y5JlWUaW5QXjFYyxWWwMMUvCltSkhJZQSgJpQ5q0\nKS1pT0NTmkDCKWlo0pQ0UCCEkBwwrSEx+2rC5gXjHa+yZUuWtVnLaBvN2z/u1TCyjSABJGf0+5yj\nM3PvXM15R/No7jz3fd735eLZ45hXOXgipmWnjOdQew93v7yb21Zuob6tmy117fzoyrkUZafxpcWT\n6I3FuePpdwiY8fiGOi6ePS4xiyjACUVZnDuthGe2NHD5KeNZMrWE+ZWF3LNqN1fOn0BaKMCru5r4\n8Qs72bD/MJPGZPH9P57DCUVZtHb1kpcRTswieqitm0fX1fLougNsb+hIvO/glckOrBnX3dfPG7ub\naersoTQnna6+ftq6+4gEg8yZkM+4vHQaO3pZX+t9gR6YRRPgQGs3+ZlhXt/VzF0v7eTt2sNMLsnm\nz86o5Lc7m5g9Po8HXtubmOTivOmlhIMB1tS0UJKbRnl+BuPyMyj0Z518bXcTkVCAPz9zInev2s3/\nvLKHB1/fy8NragdNopKb7vV0T0lKmoJm9PY79jZF+a8Xd1Jdks2s8XmJ9/wHV8zh4TW1vLy9cVC5\n3szyfO8La6yfry+dygvbDlFZlMmM8jx+umo3lUVZbKj1ym7zMyPMrsjj12/Xcdnccpav3c/kkmzC\nwQD/+9YByvLSuWL+BF7b6VUh/NW5k/nR8zu4/9UaLp49jqKsCOUFGfT1O9bWtNDW5ZWwJao2Ono4\nHO2j3zl+8cZecjPCVI/JZtWOQ9S2RDmzupii7DQ2fet8QsEA6/a14hzc+9s9TC/LZdrY3ETJ43++\nsJP/fnk3lUWZ5GWEae+O8caeZhZWFdHY0cOXHlgLwPf+aBbffWIr9/52N+D14FUUZPDiO4d46I29\njMvPwMyb0bcwK4KZV6L9+Po6ntpcz5nVxVx52gR2Herk9ie3cf5JY/nOZ2bQ1hVj+dpaxhdkcvqk\nInY3dfL4+jrM4OLZ3mfVOwe9EutY3HHL45uZXZFPJBRI/E/MrcgnPRxk2SnjuXnFJp7dcpBbHt/M\nnqYoX3lwHSu/ehYFWRGcc6yvbSUYMKaUDk6kBxK8mqYo976yh2e3NvBmTQu/vG4BPX1xcjNCv/ek\nGs9tbWBqaQ4dPTGe2fLRJHiDeso/wFjZeNyxua6Nz971Kt19/Vy/eBLTy3K5beVWor0xbrlsBp09\nMfIzI3T2xFi5sZ5QwNjbHOXHf3IyT2yq5/Ynt1F/uJtzppVQUZjJofYeQkFj1vg8emNxWqN9vOJP\nenRkiWY87jA7emKSnpj3OTIwpnNA9ZjsxDm/ojCDioJMNh1oo7ali4nFWYwvyOCRtfv5/OmVFGR6\n723yRYPknsTk9fjau/v4i5+vobali66+fu7wh0/saGjnm/+3idZoH9PKcvjCmVX09ccZl5/BmJw0\nfvT8DgIB4wtnVTHwCv5QSzQ/zgSvHEgeCVsLnPZexzjnYmZ2GCgChp4uS+QPWEVh5lFXx4IB40uL\nJx3z+MqiTH527Xw2HWjj9ElFhIIBvv3pmQBcOLOMv146ZdC4xYnFWXxv2axBE6ocy+dOrWDx5DGJ\nEq4jLagq4gdXzBm0thzAJ2eWceGMsaza0chDb+6joa2bgswIJ08oICMS5HOnTiA7PURaaAu7Gjt5\n6M29hAMBfnjlXJo6vNKhYMDISQvxxUVV/PC57fzkpV2JNZ+Se1O+tHgSpblpzK0o4JI54wa9zvkT\nB3+hPtLl8yqYUprD3at289l5FZw52Ru3V5Kbzl1XzeOul3Z6J/lr5jN9XC63fmYm33psE1ctOIGS\n3PRBXwwmFGZSlBXhe8tmkZcZ5pbLZnDns9u5blEV5580uEwyLzPM0umlTCnN5qL/WIVhPPCF+Zw8\noYAHqmqIO/j3P57znle2zSzRy/flJdVcv7hqyNf5QSQnYX+64IREr12yJdNKWPtPSznU3kNlcWai\np+2CGd7Fk6y0ECvWH+DEsblcOHMsS6aWDHpe8F7X3Ip8+vodf3ZGJa/saBpUIlWSk05R9hHlj9ne\nNOxfP28Kf3Ty0Rci3svM8jyqS7L5t8tnY+atB3bdoir++bHN/NU51aRH3l1C4tbPzDrq988/aSxP\nfO0sHl5d+ztfICrO9kru/vXXWzhnWgm3Lzv6+cEreb5wZhmPrT9AfmaYn1x1yqCy2huWVPPmnmaW\nr9vP6ZOKuPni6Uc9x/WLJ7FqRyNXn14JeBdZvnj/aqbf/ASGV8I1Li+dC04ayxOb6ll8+wuEg0Zf\nv2PSmCzSQt7f4c7ndgB448E6eylIKkMNBow397Sw9I4XqWmKDiqbPtZrau7sHbxMgv/58Ik7Xkwc\nV1mUydc+MZl7Vu3mHx/dSH5mmF+/XUdZXjp3fHY2uxs7ufeVPQQCxtlTx9AS7eOdg+28sO1QYla8\n4pwIVy88gfL8DK44tYJf+jMoX3vmRK9HPieNqjHZ/Gr1Pp7cVD+oBzgjEmRLXRuLbn+ejHCQOz83\nl5KcNJ7adJCbLpzGpXPKyYyEeHl746AevEgowBnVRTR39jJzfB43LKlmybQxVBRksrc5yt898jbg\nTXYVCQW49dOz2Fb/Clff8wYAN54/lfRwkM2Pb+aWS2eQnRbirCnF/OCKOXxqZhkbalu5/9WaRKWD\nmZeMDvSgnlFdlFjf7q4Xd3HXi7uOuT7cRbPK+Mq53nlk4LNkxrg8MsJBapqi3HLZjMRjVcVZBAPG\n7Ip89jVHqTvcTVZakC+eNZEbz5+GGfzn8ztZs7eFS+aMoyQ3jdtWbmVPUydjc9O58YJpbK1v56bl\nG94zLrIiQRZPHcOL2w6x0i/lnjY2h0fW1vLExjq6Y/FjJienTyo66gLLjedP5fYnt7G+tpXy/AxK\n/eT/PH8W3Utmj+Pbv97CtfetJhw0brn0JG55fAvzv/MMGeEgHT0x4s4bSnHkmP7S3HQiwQDfXbmV\n9p4YF84Yy8qN9Zx085P0xx3p4QBleRlkp4WIO28Cj4E+h4FJbXpjcXr9tVUjQW925kDAq/q4flEV\nHT0xHnx9L+d9/0UCZqSFg4QCxoHWLn/YRtqgWRqdA4c7at1Ds3cnVwv6y/HsaYoy71+foTArTCTk\nVcD0x72fxo4eOnv7CQaM4qwI804sTazLWlGYQXF2Gjc8OHh8YHF2Go0dPZw1uZgLZoxlqX/R5aE3\n9/Kz12re8/0Gbymb6pJsVm1vpLmzlyn/uJLe/jhZkSAFWREOR/vIywwT63fU+yWryWu1fuWcaq45\nozLR83ni2FxOKMrkp6u8iwvXL5rEqZUFXHvfahZ851mc/3eoKs7COS+5GygdzogEyUoL0dXXz4yb\nnyQ97JVOf2pWGcvX7mfd3lbCQe/vl50WYk5FPk9srGf52v2Jv+/kkmzeOdjO5xdWUuovxh4JBSg4\noqrqD4V9XJ1lZrYMuMA59wV/+yrgNOfcDUnHbPSPqfW3d/rHNB7xXNcB1wFMmDDhlJqaoYNORI4f\nff1x4s4dVZqXLHkygWNd/Rtpzjl6YvEhJwA6lqaOHrLSQonfc84dd69tODW0dZMeCQ4qE3LO0dYd\ne9/xrB+Ec46apiiVxVnvf/CH0BPrT4zvPLWycFCCcKSOnhj7W7qoLsk+5mxs7d19rKlpYdHkMe+5\nBuiRY3Of39bA+n2t9McdFQWZiYsf+1u7eHh1LV19/eRmhFhYVcTcCQU8tame3Y2dVBZnsfTEUt7Y\n00xFYWaih3PF+gP85u06+p2jqjiLBVVFVBRm0NDeQ1bEW4qjvbuPdXtbWb+vldK8dGaW57Gqf02p\nAAAKtElEQVTrUAdXLawk2hvjF6/vZUxuOvG4ozw/gyXTSggGjH3NUfY1R1k4qYjdjZ2U5KYnEu/u\nvn7MBs+sOtT/Wt3hLnr64sd8f1ujvURCgURP68G2bp7ZcpDmjl4un1eRSCKOLBvcUHuYaWU5g97D\n9u4+4vF311BMbturO5sIhwLMrchPPM/Btm5WbqijKDuNpdNLCQaMrXXticmrktU0dfLSO4cYX5hJ\nS2cve5uj9MTinDaxkJ2HOplZnsf8iYVsrW9jR0MH+1u6aOrs5fJTxuOAusPdTBub856TEMXjjpZo\nr9+7NpBAvLukxQflnKMvaW3B7r5+ntva4JewG82dPTR39hF3jgmFmcyfWEh6OEhrtJfNdW045yVv\nz2xp4OXth8hND3PZ3HJ2N3ay81AHVcVZtEb7OK2qMDFr7du1rYQCAaaUZnPnczuobYly9tQSLpk9\nzlt8Ounc8NSmehrae1hQVUh1SQ6/3dnIS+800tXrlQHmZYQ5vbroqDF4APes2s1b+1qZUprNl5dU\n88BrNextjlKSk05DezcHDncT7Ykl1iANmHchJdbvrTsb8dcf7O33Er2emLdmal5GmG9eNJ1Y3HHn\ns9vpjcXp9+O5LxanLM9LGpo7e3F4Jc0D74iZYbw7JGPgq7nDuyjzwyvn8vquZh5ZW0tmJEhzZy+x\nfm9B7oG1WAszI+Skhzjc1cc1Z1RSXZLDgdYuapqizByfh+GNKZ1cmkNTRw+xuGNhVRF1bd3kZ4QH\nlTFGe2Ns3N/G/tYoxdlpRHv72Xygjay0IHkZYSqLsjjVr0La0dDOg6/vIxIKkB4OcLirL3ERqSXa\nSzBgVBZlkRYKcNncckpz0xPjUwdi61uPbeZvlk6hq7ef+17dQ9w5rltURVleBjsaOnhkbS0B88Z4\ne0skBbxxd+EgBZlhvrykms7eGMvX7qf+cDeH2ns498QSLp1Tzo9f2MGWunZi8ThleRn85dmTKMlN\np7Gjh5Ub6xmTHWHdvla21bczuSSbG5ZMTvzv17ZEGZubPmSp8UgyszXOuWOuAfZxJngLgX92zp3v\nb38DwDl3a9IxT/rHvGpmIaAeGDNUiea8efPc6tWr3+thERERERGRlDZUgvdxpqRvApPNbKKZRYAr\ngBVHHLMCuNq/vwx4TuPvREREREREfj8f2xg8f0zdDcCTQBC4xzm3ycz+BVjtnFsB3A38zMx2AM14\nSaCIiIiIiIj8Hj7OSVZwzv0G+M0R+76ZdL8buPzjbIOIiIiIiMhocXyOGhQREREREZHfmRI8ERER\nERGRFKEET0REREREJEUowRMREREREUkRSvBERERERERShBI8ERERERGRFKEET0REREREJEUowRMR\nEREREUkRSvBERERERERShBI8ERERERGRFKEET0REREREJEUowRMREREREUkRSvBERERERERShBI8\nERERERGRFKEET0REREREJEWYc26k2/A7MbNDQM1It+MYioHGkW6EHLcUHzIUxYcMRfEh70cxIkNR\nfKSmE5xzY471wB9cgne8MrPVzrl5I90OOT4pPmQoig8ZiuJD3o9iRIai+Bh9VKIpIiIiIiKSIpTg\niYiIiIiIpAgleB+dn4x0A+S4pviQoSg+ZCiKD3k/ihEZiuJjlNEYPBERERERkRShHjwREREREZEU\noQTvI2BmF5jZNjPbYWY3jXR7ZPiZ2T1m1mBmG5P2FZrZ02a23b8t8Pebmd3px8vbZnbyyLVchoOZ\nVZjZ82a22cw2mdlX/f2KEcHM0s3sDTNb78fHt/z9E83sdT8OfmlmEX9/mr+9w3+8ciTbL8PDzIJm\nts7MHve3FR8CgJntMbMNZvaWma329+n8MoopwfuQzCwI/Ai4EJgOfM7Mpo9sq2QE3AtccMS+m4Bn\nnXOTgWf9bfBiZbL/cx3w42Fqo4ycGPB159x0YAHwZf9zQjEiAD3AOc652cAc4AIzWwB8F/i+c64a\naAGu9Y+/Fmjx93/fP05S31eBLUnbig9JtsQ5NydpOQSdX0YxJXgf3nxgh3Nul3OuF3gIuHSE2yTD\nzDn3EtB8xO5Lgfv8+/cBlyXtv995XgPyzaxseFoqI8E5V+ecW+vfb8f7klaOYkQA/33u8DfD/o8D\nzgEe9vcfGR8DcfMwcK6Z2TA1V0aAmY0HPgX81N82FB8yNJ1fRjEleB9eObAvabvW3ydS6pyr8+/X\nA6X+fcXMKOaXS80FXkcxIj6//O4toAF4GtgJtDrnYv4hyTGQiA//8cNA0fC2WIbZvwN/B8T97SIU\nH/IuBzxlZmvM7Dp/n84vo1hopBsgMho455yZacraUc7MsoFHgK8559qSL6orRkY351w/MMfM8oFH\ngWkj3CQ5TpjZRUCDc26NmZ090u2R49KZzrn9ZlYCPG1mW5Mf1Pll9FEP3oe3H6hI2h7v7xM5OFD2\n4N82+PsVM6OQmYXxkrufO+eW+7sVIzKIc64VeB5YiFc6NXAhNjkGEvHhP54HNA1zU2X4nAFcYmZ7\n8IaBnAP8AMWH+Jxz+/3bBrwLRPPR+WVUU4L34b0JTPZns4oAVwArRrhNcnxYAVzt378a+L+k/Z/3\nZ7JaABxOKqOQFOSPf7kb2OKcuyPpIcWIYGZj/J47zCwDWIo3TvN5YJl/2JHxMRA3y4DnnBa1TVnO\nuW8458Y75yrxvmM855z7ExQfAphZlpnlDNwHzgM2ovPLqKaFzj8CZvZJvPr4IHCPc+7bI9wkGWZm\n9gvgbKAYOAjcDPwv8CtgAlADfNY51+x/2f8h3qybUeAa59zqkWi3DA8zOxN4GdjAu2No/gFvHJ5i\nZJQzs1l4kyAE8S68/so59y9mVoXXY1MIrAP+1DnXY2bpwM/wxnI2A1c453aNTOtlOPklmn/rnLtI\n8SEAfhw86m+GgAedc982syJ0fhm1lOCJiIiIiIikCJVoioiIiIiIpAgleCIiIiIiIilCCZ6IiIiI\niEiKUIInIiIiIiKSIpTgiYiIiIiIpAgleCIiMmqZWb+ZvWVm681srZmd/j7H55vZX36A533BzOZ9\ndC0VERH5YJTgiYjIaNblnJvjnJsNfAO49X2OzwfeN8ETEREZKUrwREREPLlAC4CZZZvZs36v3gYz\nu9Q/5jZgkt/rd7t/7N/7x6w3s9uSnu9yM3vDzN4xs7OG96WIiMhoFRrpBoiIiIygDDN7C0gHyoBz\n/P3dwKedc21mVgy8ZmYrgJuAGc65OQBmdiFwKXCacy5qZoVJzx1yzs03s08CNwOfGKbXJCIio5gS\nPBERGc26kpK1hcD9ZjYDMOA7ZrYIiAPlQOkxfv8TwP8456IAzrnmpMeW+7drgMqPp/kiIiKDKcET\nEREBnHOv+r11Y4BP+renOOf6zGwPXi/f76LHv+1H51sRERkmGoMnIiICmNk0IAg0AXlAg5/cLQFO\n8A9rB3KSfu1p4Bozy/SfI7lEU0REZNjpiqKIiIxmA2PwwCvLvNo5129mPwceM7MNwGpgK4BzrsnM\nXjGzjcBK59yNZjYHWG1mvcBvgH8YgdchIiICgDnnRroNIiIiIiIi8hFQiaaIiIiIiEiKUIInIiIi\nIiKSIpTgiYiIiIiIpAgleCIiIiIiIilCCZ6IiIiIiEiKUIInIiIiIiKSIpTgiYiIiIiIpAgleCIi\nIiIiIini/wH2ISQLlFredQAAAABJRU5ErkJggg==\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3gAAAHwCAYAAAD0Es3SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde7Qs51nf+d9T3b330ZGOfNORDb5gY2wT5zbJCEhCJrGTeLCTjM0aZjJ2EgayhjghMbnAMIHMLAecmQmBhCSsMVl2uEyYQAwDgQgiYgI2tpnYjmRMMLItW5YtS7IsnXN0Ode9d3fVO39UvVVv1e5LVXdVX6q/n7Wk3rt3X6r79KWeep73ecw5JwAAAADA7os2vQEAAAAAgHYQ4AEAAABATxDgAQAAAEBPEOABAAAAQE8Q4AEAAABATxDgAQAAAEBPEOABAPaCmf2ymX1T25dtuA2vMrOH275dAAC84aY3AACAWczsavDrWUnHkuLs97/inPvJurflnHtdF5cFAGCbEOABALaWc+4W/7OZfU7StzjnfrV6OTMbOucm69w2AAC2ESWaAICd40sdzezvmNkXJf24mT3LzH7JzC6Y2ZPZzy8IrvPrZvYt2c/fbGa/YWb/KLvsZ83sdUte9iVm9n4zu2Jmv2pmbzezf1Xzcfyu7L6eMrN7zez1wd/+tJl9PLvdR8zsf87Ovy17bE+Z2RNm9gEz4/scACCJAA8AsLueJ+nZkr5M0puVfqf9ePb7iyTdkPR/zbn+10i6T9Jtkr5f0o+amS1x2Z+S9J8kPUfS90j6xjobb2YjSb8o6Vck3S7p2yT9pJm9IrvIjyotQz0n6fdIek92/ndIeljSeUnPlfR3Jbk69wkA6D8CPADArkok/T3n3LFz7oZz7pJz7uecc9edc1ck/R+S/vic6z/onPsXzrlY0r+U9CVKA6balzWzF0n6Kklvdc6dOOd+Q9KdNbf/D0m6RdL3Zdd9j6RfkvSm7O9jSa80s1udc086534zOP9LJH2Zc27snPuAc44ADwAgiQAPALC7LjjnjvwvZnbWzN5hZg+a2WVJ75f0TDMbzLj+F/0Pzrnr2Y+3NLzsl0p6IjhPkh6quf1fKukh51wSnPegpOdnP3+DpD8t6UEze5+Z/eHs/B+QdL+kXzGzB8zsu2reHwBgDxDgAQB2VTVr9R2SXiHpa5xzt0r6Y9n5s8ou2/CopGeb2dngvBfWvO4XJL2wsn7uRZIekSTn3N3OuTcoLd/8BUk/k51/xTn3Hc65L5f0eknfbmZ/csXHAQDoCQI8AEBfnFO67u4pM3u2pL/X9R065x6UdI+k7zGzgyzL9t/UvPqHJV2X9L+Y2cjMXpVd913Zbf0FM3uGc24s6bLSklSZ2Z81s6/I1gA+rXRsRDL9LgAA+4YADwDQF/9U0k2SLkr6kKR/v6b7/QuS/rCkS5L+d0k/rXRe31zOuROlAd3rlG7zD0v6H51zn8wu8o2SPpeVm/7V7H4k6WWSflXSVUkflPTDzrn3tvZoAAA7zViXDQBAe8zspyV90jnXeQYRAIAqMngAAKzAzL7KzF5qZpGZvVbSG5SumQMAYO2Gm94AAAB23PMk/Rulc/AelvStzrmPbnaTAAD7ihJNAAAAAOgJSjQBAAAAoCcI8AAAAACgJzpdg5ctNv9nkgaSfsQ5932Vv/8TSa/Ofj0r6Xbn3DPn3eZtt93mXvziF3ewtQAAAACw/T7ykY9cdM6dn/a3zgI8MxtIeruk1yhddH63md3pnPu4v4xz7m8Hl/82SX9g0e2++MUv1j333NPBFgMAAADA9jOzB2f9rcsSza+WdL9z7oFsmOu7lLaOnuVNkv51h9sDAAAAAL3WZYD3fEkPBb8/nJ13ipl9maSXSHpPh9sDAAAAAL22LU1W3ijpZ51z8bQ/mtmbzeweM7vnwoULa940AAAAANgNXQZ4j0h6YfD7C7Lzpnmj5pRnOufe6Zy7wzl3x/nzU9cSAgAAAMDe6zLAu1vSy8zsJWZ2oDSIu7N6ITP7SknPkvTBDrcFAAAAAHqvswDPOTeR9BZJ75b0CUk/45y718zeZmavDy76Rknvcs65rrYFAAAAAPZBp3PwnHN3Sbqrct5bK79/T5fbAAAAAAD7YluarAAAAAAAVkSABwAAAAA9QYAHAAAAAD1BgAcAAAAAPUGABwAAAAA9QYAHAAAAAD1BgAcAAAAAPUGABwAAAAA9QYAHAAAAAD1BgAcAAAAAPUGA14Kf/PCD+vu/9PHWbu/Rp2/o937Pu/Xpx660dpsAAAAA+o8ArwW/9fmn9Msfe7S123v06SNdOZrooSevt3abAAAAAPqPAK8Fh6NIx5OktdtzrnwKAAAAAHUQ4LXgzHCgo3Hc2u25LLIjwAMAAADQBAFeC1rP4FVOAQAAAKAOArwWHA4HmiROk7idIC9J0tAuIYUHAAAAoAECvBacGaVPY1tZvDyDR3wHAAAAoAECvBYcDgeS2gvwiswdER4AAACA+gjwWuAzeK01WqGLJgAAAIAlEOC1oP0MXnpKfAcAAACgCQK8FrSdwXOiyQoAAACA5gjwWtBZBo/4DgAAAEADBHgtOBy2nMHzg85buTUAAAAA+4IArwWHo3YzeC7P4BHiAQAAAKiPAK8FrWfwyN0BAAAAWAIBXgvOtJzBS7KbockKAAAAgCYI8FoQZvD+u3/+H/WO931mpdvLx5wT3wEAAABogACvBWEG7+OPXtYDF66tdHs+c0eABwAAAKAJArwWHGZz8K4fT3T9JNY4Xq1U0zHoHAAAAMASCPBacCabg/fEtRNJ0snKAZ4rnQIAAABAHQR4LRgNTGbSxatpgLdyBs+fEt8BAAAAaIAArwVmpsNhpEvXjiVJ43i1yCxfg0eRJgAAAIAGCPBacmY00KW2MniufAoAAAAAdRDgteRwGOnSVZ/BWy3AKzJ4AAAAAFAfAV5LzowGunjNZ/DaCc3I4AEAAABoggCvJYfDSCeTNHPXVgYvIcIDAAAA0AABXkv8sHNJeaC3LObgAQAAAFgGAV5LDofFU7l6Bi/7gQweAAAAgAYI8FoSZvAmyWqBmaPJCgAAAIAlEOC1pJTBa6tEkwgPAAAAQAMEeC05HAZr8FbsoukHnNNkBQAAAEATBHgtORy1vwaP+A4AAABAEwR4LQkzeKsGeHTRBAAAALAMAryWnAkyeJMVSzR9aaYjhQcAAACgAQK8lpTX4CUrBWeEdQAAAACWQYDXkjCDJ602KiEfk0CkBwAAAKABAryW+AyeWfr7KuvwfGBHF00AAAAATRDgtcTPwXvmTSNJ0niyfHCWMOgcAAAAwBII8FpyZpRm8J519kBSug5vWQw6BwAAALAMAryW5Bm8s2kGb5IsH+AVGTwiPAAAAAD1dRrgmdlrzew+M7vfzL5rxmX+nJl93MzuNbOf6nJ7uuQzeM++Oc3grVKi6ZHBAwAAANDEsKsbNrOBpLdLeo2khyXdbWZ3Ouc+HlzmZZK+W9LXOueeNLPbu9qervkMXhslmszBAwAAALCMLjN4Xy3pfufcA865E0nvkvSGymX+sqS3O+eelCTn3OMdbk+nXnzbzTp3ONTLn3tOUjtdNInvAAAAADTRZYD3fEkPBb8/nJ0Xermkl5vZ/2dmHzKz13a4PZ36ittv0ce+9+v0kttulrRagOdH6BHfAQAAAGiisxLNBvf/MkmvkvQCSe83s9/rnHsqvJCZvVnSmyXpRS960bq3sZFRVqo5jlcYdC4GnQMAAABorssM3iOSXhj8/oLsvNDDku50zo2dc5+V9CmlAV+Jc+6dzrk7nHN3nD9/vrMNbsNokE46b6VEkxweAAAAgAa6DPDulvQyM3uJmR1IeqOkOyuX+QWl2TuZ2W1KSzYf6HCbOncw8Bm8VQK8NLBLiO8AAAAANNBZgOecm0h6i6R3S/qEpJ9xzt1rZm8zs9dnF3u3pEtm9nFJ75X0nc65S11t0zoMWwjw8sCOGk0AAAAADXS6Bs85d5ekuyrnvTX42Un69uy/XvAlmicrzMFzNFkBAAAAsIROB53vI1+iOUnamIPXyiYBAAAA2BMEeC0btbEGLz8lwgMAAABQHwFey/IxCSuVaNJkBQAAAEBzBHgty9fgtTEmgQAPAAAAQAMEeC0bRW100czW4FGiCQAAAKABAryW5SWaLazBI74DAAAA0AQBXst8ieY4Xj46KzJ4AAAAAFAfAV7L2ijR9JFdQpcVAAAAAA0Q4LUsikzDyFpagwcAAAAA9RHgdWA4sJVKNOmiCQAAAGAZBHgdGA0inUxWyeClp3TRBAAAANAEAV4HDgaRJskqXTSzEk3iOwAAAAANEOB1YDSINJ60UaJJhAcAAACgPgK8DoyGqzVZcTRZAQAAALAEArwOjKJIJyt10UxPSeABAAAAaIIArwOjQbRaBs+vwSOHBwAAAKABArwOjIamyQpjEsjgAQAAAFgGAV4HRoPVSjTzJistbQ8AAACA/UCA14FVSzSVj0kgxAMAAABQHwFeBw4GkcarlGhmsSHxHQAAAIAmCPA6MBysOCaBQecAAAAAlkCA14HRINLJpIUxCazCAwAAANAAAV4HDgaRJsnywZmjiyYAAACAJRDgdWC0aolmFtmtECMCAAAA2EMEeB0YDSKNVyjRdPkpER4AAACA+gjwOjAcRDrJumj+8K/fr5/7yMONrp8wCA8AAADAEgjwOnAQlGj+3Ece1r/9z19odH3iOwAAAADLIMDrQDjofJI4PX39pNH1fQaPQecAAAAAmiDA68BwEGmSlWhOYqcnr48bXd+HdTRZAQAAANDEcNMb0EejgWmcpBm8cZzoylHc6Po+c0d8BwAAAKAJMngdGEaRnJPixGmSOF0+mihukI4r5uAR4gEAAACojwCvA6OhSUqzd34t3tM36pdpJmTwAAAAACyBAK8Doyh9Wsdxkq/Fe6pBoxVXDMIDAAAAgNoI8DowHKQZvEnsNMnW4jVptOKrORNKNAEAAAA0QIDXgeEgy+AlicZZBu/pG01GJfgxCW1vGQAAAIA+I8DrwChKM3jH4yQ/76klMniOGk0AAAAADRDgdWCUZfCunxTjEZqUaOZjEojvAAAAADRAgNcBvwbvxrgI8J5u0GSlyOABAAAAQH0EeB3wGbwby2bw/CkpPAAAAAANEOB1YJitwTsKMnhPNZiDR4kmAAAAgGUQ4HUgz+CFAd4Sc/CI7wAAAAA0QYDXgWklms26aPoMHiEeAAAAgPoI8Drgm6xczzJ4w8j0VIM5eGTwAAAAACyDAK8DoyzAO8oyeLfdcrhUBi8hwgMAAADQAAFeB4ZReQ3ebecOdOVookmczLtaji6aAAAAAJZBgNeB6hy8Z9w0Kv2+CIEdAAAAgGUQ4HXgoNJk5ezBUJJ0MqmZwXPlUwAAAACogwCvA8NKgHfLYRbg1SzRzLto0mYFAAAAQAMEeB3wg859F82zBwNJDTJ4/pT4DgAAAEADBHgdqM7Bu/mwWYmm756ZEOEBAAAAaKDTAM/MXmtm95nZ/Wb2XVP+/s1mdsHMfiv771u63J518U1WjioZvOOaAZ7yQeftbxsAAACA/hp2dcNmNpD0dkmvkfSwpLvN7E7n3McrF/1p59xbutqOTcgzeFmAd3PWZKVugJcw6BwAAADAErrM4H21pPudcw84504kvUvSGzq8v63hB51fX7JEM2+uQoQHAAAAoIEuA7znS3oo+P3h7LyqbzCz3zaznzWzF3a4PWvjB537Es2bD7MmK3W7aGYXo4smAAAAgCY23WTlFyW92Dn3+yT9B0n/ctqFzOzNZnaPmd1z4cKFtW7gMnwGL2+y0nQOXnaaEN8BAAAAaKDLAO8RSWFG7gXZeTnn3CXn3HH2649I+i+n3ZBz7p3OuTucc3ecP3++k41tk5lpEFm+Bu/sYcMxCXmTFSI8AAAAAPV1GeDdLellZvYSMzuQ9EZJd4YXMLMvCX59vaRPdLg9azUa2KkmKydxXOu6jiV4AAAAAJbQWRdN59zEzN4i6d2SBpJ+zDl3r5m9TdI9zrk7Jf0NM3u9pImkJyR9c1fbs26jKNKV44mkYg3e8bhuF03GJAAAAABorrMAT5Kcc3dJuqty3luDn79b0nd3uQ2b4mfhSdLZPIPXbA0e8R0AAACAJjbdZKW3hoPiqW3aZCVhDR4AAACAJRDgdWQUpRk8M+lwlD7NdQed52PwiO8AAAAANECA15HRMH1qR1Gkw+znxhm8BUWa7/pPn9cP/dqnV9hKAAAAAH1CgNeRYZbBGw5MZqaDQdR8Dd6CDN6vfuJx3fWxR1fYSgAAAAB9QoDXkVG2Bs8HegfDqPUums45yjgBAAAA5AjwOuK7aPpA72AYNZ6DlyyI3hLnFpZxAgAAANgfBHgdGUZZBi8L9A4GUe01eHWzcolL/wMAAAAAiQCvMwd5iWaQwasd4NUr0UycY5QCAAAAgBwBXkeKEs1iDV7dJis+K7eo/DIt0QQAAACAFAFeR/ygc78G77BJBk81M3gJs/IAAAAAFAjwOjLKxyQUJZp1B50nTZqsEOEBAAAAyBDgdeRUieagfoDn8hLNxZcjvAMAAADgEeB1ZNocvKZNVhZFb7FzC7N8AAAAAPYHAV5H8gBvqTV45dNZEgadAwAAAAgQ4HXEZ+6W66Lpm6wsWoNHkxUAAAAABQK8jgyrc/CWGHS+aIi5o8kKAAAAgAABXkdGlSYrh8NB7QAvz+AxBw8AAABAAwR4HRlVM3jDSMeTuN6VfY+VRU1WksWjFAAAAADsDwK8jvgxCcNB8y6axRq8+ZdzNFkBAAAAECDA68goy9yNgkHndZus1I3ZKNEEAAAAECLA60iewYuKQefj2ClZ1DlFTbtoEuIBAAAASBHgdaQ6B+9gmJ7WyeLV7aLJHDwAAAAAIQK8jpzuotk8wFvYRTOhRBMAAABAgQCvI757pj/NA7wajVZ82LYoO5c4umgCAAAAKBDgdaSawfMlmsc1Arwkz+AtuhwlmgAAAAAKBHgd8WvvRtU1eHUyeLXHJNBkBQAAAECBAK8jvntmPgdvMJBUt0QzO13YRZMMHgAAAIACAV5HfMZuuQxedrrgcjFNVgAAAAAECPA6UjRZKa/BO4njudcLs3bMwQMAAADQBAFeR/JB5z6DN6jXZCWcfbcodHPOLZyVBwAAAGB/EOB15NQcvFG9AK+cwZt/H4lzC2flAQAAANgfBHgdqc7B8xm8RWvwwozcohl36Ry8FTYSAAAAQK8Q4HXEN1fxpZp1B52XMnI1Mngk8AAAAAB4BHgdmTXofGGA12ANXpJQogkAAACgQIDXEd9cJS/RzLtoNgjwKNEEAAAA0AABXkduOUwHm9+cnfo1eOM40d2fe0J/9B++R9eOJ6euF667W5jBc44xCQAAAAByBHgd+Yrbz+nH/9JX6Y+97LwkaRSUaH7qsSt6+MkbeuLayanr+XDNbHGTFZbgAQAAAAgNN70BffbqV9ye/5x30YwTDSfpurxpAZw/b2BWb0wCER4AAACADAHemoyCMQnDyAd4py/nA7YosoXZuTi7sHNOZtbWpgIAAADYUZRorskgMg0i0zhONI7TwGxaBs8FGbx5EZ4Lsndk8QAAAABIBHhrdTCIdDJJdJyNSpjWICXP4JnmjkAIr7porR4AAACA/UCAt0ajgWkcO42zUQnTJib4YC2KbO4IhCbdNgEAAADsB9bgrdHBcKCTONFgTpMVf05kNncEQhj8kcADAAAAIBHgrdXBwCpNVuZ00VzQZCW8LiWaAAAAACQCvLUaDSON40QnWQZvalyWr8GbPyaBoA4AAABAFQHeGvkmK4MsgxdPWWTnz8qmKswcgUCJJgAAAIAqArw1Gg3SDN78NXhZkxUrsnzTRtxRogkAAACgigBvjQ6GkU5ip0GUts+c1iUzCUo0pdkdMl3QgZPwDgAAAIBEgLdWaYlmrEGWkZs+B69oslL8fjqFF5PBAwAAAFDBHLw1Gg39HLw0IJu2Bs/la/DmZ/BKc/CI7wAAAACo4wDPzF5rZveZ2f1m9l1zLvcNZubM7I4ut2fTfJOVk8nsEk0frJmVf68qZe0I8AAAAACowwDPzAaS3i7pdZJeKelNZvbKKZc7J+lvSvpwV9uyLXyTlZM4DfCmlWjmc/BsdiOW9LqnrwMAAABgv3WZwftqSfc75x5wzp1IepekN0y53N+X9A8lHXW4LVvhYFgjg5edRtNaZwZKJZptbSAAAACAndZlgPd8SQ8Fvz+cnZczsz8o6YXOuX8374bM7M1mdo+Z3XPhwoX2t3RNDgaRTuJE4yyDF8/J4EXRnGHoqs7BI8QDAAAAsMEmK2YWSfpBSd+x6LLOuXc65+5wzt1x/vz57jeuI9USzalz8KqDzmfk55Ik7KLZ7nYCAAAA2E1dBniPSHph8PsLsvO8c5J+j6RfN7PPSfpDku7sc6MVX6I5nsxeg+cqa/DqNFmZFQQCAAAA2C9dBnh3S3qZmb3EzA4kvVHSnf6PzrmnnXO3Oede7Jx7saQPSXq9c+6eDrdpo9IMnisyeMnpy/hQzRYMOi9l7YjvAAAAAKjDAM85N5H0FknvlvQJST/jnLvXzN5mZq/v6n63WbXJyrw1eH4O3qwOmYmjRBMAAABA2bDLG3fO3SXprsp5b51x2Vd1uS3b4GBgOokTRX593Zw1eNGCOXiOEk0AAAAAFRtrsrKPDobp0300nj0mIe+imU86n35bcVDeSRNNAAAAABIB3lqNBuWne34XTb8Gr06JJhEeAAAAAAK8tTod4J2+zKkAr04XTeI7AAAAACLAWytfouklUyI8n7HzXTRnZecI6gAAAABUEeCt0UGNEk0f8w3mL8GjRBMAAADAKQR4a3Qqgze1RLM8JmFW7BYn5RLNb/zRD+sXPvrI9AsDAAAA2AsEeGtUp8mKj9siW9RkpXw7H/7sE7r3C0+3s6EAAAAAdhIB3hqNfN1lZtoaPF+UuWhMQnkOXvo7A88BAACA/UaAt0Z1SjSTShfNWUFbeL5z6X8xER4AAACw1wjw1qhOkxV/VpHAWzwHL83eOZqtAAAAAHuOAG+Nqhk8N3UNXr0mK2F5p1Oa0SPAAwAAAPYbAd4aVZusTCupzAed501Wpqs2WUlvb+VNBAAAALDDCPDW6HQXzdOX8Vm9KM/gLS7R9IHi9KYtAAAAAPYFAd4anW6yMiWDl51Gfg3ezCYrwaDzLHMXU6IJAAAA7DUCvDWqNlmZFo/VXYMXnu8Du21Yg/c33/VR/dvfYuA6AAAAsAkEeGtUzeBNy7j5sxYPOg9LNNMU3jaUaL7nk4/rNx98ctObAQAAAOwlArw1OjXofIUummGDFv9jvPn4Lp3HtwWZRAAAAGAfEeCt0ejUmITTlynW4NXvopk3WdmCwCqdybfprQAAAAD2EwHeGp0adD51TELWRTML8GYFbW5Lu2gmbnbnTwAAAADdIsBbo2qAN38NXvn3qmkZvGlz9dYtcW4rtgMAAADYRwR4axRFpmFUrMObFgf58wb55Wo0Wcm7aLaymStxbju2AwAAANhHw01vwL4ZDSKZJYoTN7WU8fSg8+m3U56Dtz1r8BLntqJUFAAAANhHBHhrNhqYzCI5l8zoopmeDhY2WTm9Bm8bSiMT57Yi0AQAAAD2Ua0STTO72cyi7OeXm9nrzWzU7ab108FwoINhpMhMcTLtEr7JSvrbrGApCa67VV00RYkmAAAAsCl11+C9X9IZM3u+pF+R9I2S/u+uNqrPDgaWlWlO7zbpg6MmJZrFGrzNRlbOOebgAQAAABtUN8Az59x1Sf+tpB92zv33kn53d5vVXwfDSAeDSIPIpgZkRRfN+QFeeP62lGj6bWJMAgAAALAZtQM8M/vDkv6CpH+XnTfoZpP6bTSI8hLN6V000zN9F003YxVemCVLtqSLpt+OTQeaAAAAwL6q22Tlb0n6bkk/75y718y+XNJ7u9us/vJdNM2mB0L+nEUZvHKTley8DQdW/u6J7wAAAIDNqBXgOefeJ+l9kpQ1W7nonPsbXW5YXx0MI0VJGsDNG5MwWJBbDYMoH9hteu2bDzop0QQAAAA2o24XzZ8ys1vN7GZJvyPp42b2nd1uWj8dDCKN8jV4p/9eXYM3q3GKm9pkpd1tXRYlmgAAAMBm1F2D90rn3GVJXy/plyW9RGknTTT0mlc+V6955XMV2fTgzZ+3sEQzCUs03anz2vKpx67Uzshty1pAAAAAYF/VDfBG2dy7r5d0p3NurNkzuDHHX/5jX66/9qqvkNn8LppFk5Xp4rBEs6PmJp+/dF3/9T95v/7jZy7VunyxBo+XBgAAALAJdQO8d0j6nKSbJb3fzL5M0uWuNmofRFYeVu7lGbx8Dt7iEs1J3M0cvMtHY0nS0zfGtS5fZPAI8AAAAIBNqNtk5Yck/VBw1oNm9upuNmk/DGZl8LLTLL6bmcFLpo5JaDewajpfz+XdPFvdDAAAAAA11W2y8gwz+0Ezuyf77x8rzeZhSTZjDl7eRdPmZ/CSNZRoxg1vN98OMngAAADARtQt0fwxSVck/bnsv8uSfryrjdoHUTQ941btollnDt4kcXMvu6ykaQbPnxLgAQAAABtRd9D5S51z3xD8/r1m9ltdbNC+iBaVaC5oshJetas5eHHD2+0qkwgAAACgnroZvBtm9kf9L2b2tZJudLNJ+2EwpUTzZz/ysD7xaNq7xg86nxVbxaUxCafPa0M+X69hiSbxHQAAALAZdTN4f1XST5jZM7Lfn5T0Td1s0n6wKXPwvvcX79W144mksERz1hq804PO2y/RTE8ndUs0nT8lwgMAAAA2oW4Xzf8s6feb2a3Z75fN7G9J+u0uN67PIrNTmbE4cXn2ywd4s2KrZFqJZlcZvKYlmgR4AAAAwEbULdGUlAZ2zjk//+7bO9ievTFtDV4YoBWDzmvMwetoDV7TwDEfdM6YBAAAAGAjGgV4FdbaVuyhKDq9Bi8M+PwcvFldVqbNwWu7NLLxHDwGnQMAAAAbtUqAx178CiI7HZCFgVS+Bm/G9eMk/Hk7SjT9xQjwAAAAgM2YuwbPzK5oeoxhkm7qZIv2RFTpoumcK/2el2jOiJXC4LCrAM+XaNZtskIXTcjj9ngAACAASURBVAAAAGCz5gZ4zrlz69qQfRNZOSCrBkV+Dt6sbNj0Es12t7H5mAQ1ujwAAACAdq1SookVpGvwTgdp+d8XlGiGMVTTgeR1FZnBepdPWIMHAAAAbBQB3oZEZqWMW7W8ctBkDl5XJZoNxx44SjQBAACAjSLA25DIppdZ5n/P/mVmZvCSNZRoJqfvax5//20HmgAAAADqIcDbEDMrBUKzMnizxyQUP/tArKs5ePWbrKSnbY9rAAAAAFBPpwGemb3WzO4zs/vN7Lum/P2vmtnHzOy3zOw3zOyVXW7PNhlUSjSrw8GbNFmJsytvekxC05JOAAAAAO3qLMAzs4Gkt0t6naRXSnrTlADup5xzv9c5919I+n5JP9jV9mybKKoEabOarNTJ4FXGLbQlD9gYkwAAAADshC4zeF8t6X7n3APOuRNJ75L0hvACzrnLwa83a4+Gp6dz8GaXaEbzKzRLgVwyp9RzFUnD5i2OEk0AAABgo+bOwVvR8yU9FPz+sKSvqV7IzP66pG+XdCDpT3S4PVvFzEqZt2oZZDHofHqwNGv9Xuxca/+oTbtzNs34AQAAAGjXxpusOOfe7px7qaS/I+l/m3YZM3uzmd1jZvdcuHBhvRvYkYGVg7fTGbzFc/B8EBiWd7aZPPMBaP0xCcW2AQAAAFi/LgO8RyS9MPj9Bdl5s7xL0tdP+4Nz7p3OuTucc3ecP3++xU3cnMUlmvMzeM65vNNm1yWadcck5Gvw1hDhPXDhqk4mNSewAwAAAHuiywDvbkkvM7OXmNmBpDdKujO8gJm9LPj1z0j6dIfbs1XMrNQ5Mwz2zNL/pHlNVtzUDF6bHSzjxk1Wim3r0tM3xnrtP/2AfuGj844XAAAAAPunszV4zrmJmb1F0rslDST9mHPuXjN7m6R7nHN3SnqLmf0pSWNJT0r6pq62Z9sMql00gyDKFAR4M66fOGnoA7zguq7FpFa+Bq92ieZ6umg+dvlIJ3GiJ6+fdHtHAAAAwI7pssmKnHN3Sbqrct5bg5//Zpf3v82qJZphUBSZLRyTEDs3dVZemxm8pl00/cW6noN38cqxJGkcU6IJAAAAhDbeZGVfpQFe8fupEs3sZzcjh+ecyzN4k3h6JnBVTUs0fQav6zEJF66mAd5J3HGqEAAAANgxBHgbYjanRNNs8Rq8RFMzeK0OOvdNVmreZrEGr7VNmOrS1bQ0c0IGDwAAACghwNuQQWQzu19a8P9ZwVUSZPCqc/DasmwGr+s5eBevUqIJAAAATEOAtyHzSjSjIIM3S3kOXnF+qyWaSfl0kfCuuyzTLAI8SjQBAACAEAHehswv0QzW4M2IYVwwJiHMBLYZVyV5Rq5ehDdvrl+bLmYlmidk8AAAAIASArwNicxKwVg1g5d30ZzRZCUOBp1POhp0XoxJqHf5WV1B23bJZ/AYdA4AAACUEOBtyMCsvHYui1XOjKLyHLyZg86LEk3X0ZgEv31J3TV4pe3rPoPHGjwAAACgjABvQ6IZg87PnRllJZq+ycr064clmmGgWDcYqyNZsslKeN22OefyMQmswQMAAADKCPA2xGY0WTl3ZlgZkzC7i+bUAK/FmKco0aw5JiFIqHVVonn1eKKTrDSTNXgAAABA2XDTG7CvIquUVmYR0df97uflAYykGSvw0mAqH5PQUXOTphm8dTRZ8eWZEiWaAAAAQBUB3oYMzMqBWfbza175XP3BFz1Ljzx1I/3DjDgpdi4fdF7O4HXQZKV2gFf83NWYBD8iQZImlGgCAAAAJZRoboiZVcYbpD/7zpj5mIQZEZ4LBp0nnQV4zW7TraGL5sUraYB365khJZoAAABABQHehlTHJPhYxa+r82vwZgVKiVM+SmFbSjTDS3VWonktLdH8kmfcRIkmAAAAUEGAtyHRjEHnUZ7B8yMQpl+/3GSlfH5bmpdons5Itu14HEuSbr1pSIAHAAAAVBDgbcggKq/B88FRlP2L5F00Z5RohnPw4qB9ZatdNBs3WTl93bb5mz0cDjSesAYPAAAACBHgbUh1TIIPovI1eIsGnSfBGrwpt9OGpOGYhHWswfPbcmYUkcEDAAAAKgjwNqQ6JqHI4FVKNGdcPyzRLDVZaTGy8sFi3dsMSzTb3I5p93E4HNBkBQAAAKggwNuQqHYGb3aJph+IPpkx6Pzp62N97OGnl97GvMnKUoPOOwrwEh/gkcEDAAAAqgjwNiSKrFROmQd4UWVMwow4yTmnyNLLTZunJ0k/8cHP6X945weX3sa8yUrNeXPhpboq0fS3eziKNGYOHgAAAFBCgLchUSVDd6pE06z096rEOUVmp+bphT9fO4l1/SReulzSx0+1M3gdjWuYdh+Hw0GjDN6/+c2H9ev3Pd7JNgEAAADbggBvQ/w4BB8H5XPwsvPzAHDG9ePEKYpMkelUN86Hnrie/yxJ42S5UkaXd9Fsdvnqz21atkTzHe97QD/xwQc72SYAAABgWxDgbUiUDzIvr3PLxyQsmIPnskHnpvLA9E88eln/1fe/V7/zyNON59hV5U1Wamfwpv/cJj8eYjRoVqIZO6ejbIYeAAAA0FcEeBsS5TPsyp0qfQZPlQCwKsnW4OWL9TJfvHwkSXri2kl+28uuVVtl0HlXJZpx9rhHg0hx4upvW0KABwAAgP4jwNuQyMoZOh8c5U1WbOrVckmWwYsqlzsep2WLsXP5bU6W7DaZZxeXGHTeWRfNbO3haJg+8LplmrFzujGm6yYAAAD6jQBvQ06VaCbVOXipmYPOnZNZUcrpHU3SICYJsluTFUs0a2fjSmvwlrrLhZIkDfAOBulLt3aARwYPAAAAe4AAb0OKJivldW7FHDw/6Hx6pJSvwatk8HwQEydFBm/ZeXHNu2iG1+0qg1eswZPql5+uWqJ58eqx3vPJx5a+PgAAALAOBHgbkgd4WeyVd9GMKl00Z8QvceI0MMtvx/NBTOKCDN6Sa/CSyvrAhZevdPPsQpykmcvhYJkSzeUDvH/94c/rL//ER5YudwUAAADWgQBvQ6olmv606LFSHqNQlTinKDrVYyVfgzdJXB40TpYck9C0xLO0Bq+jJivOuUoGr26JpnTjZPkA76kbY8WJW7rcFQAAAFgHArwN8WvtqmvwihLN9HKzSjQTl5ZxnirRnEwr0Vwygxdm5GoENq6UwVvqLheKXXUNXv3s4vEkWTrwvHI0zm8HAAAA2FYEeBtipwadl7toerPn4KXjAqwS4fkMXuKKbNOyJZphc5U6a+rWUaLpu4c2zeD57TmeLJfNvHI0kdTd+AcAAACgDQR4GzLIxyQUJZoWBGyLxyS46U1W8gxekXUbL1ui2XCunVtDiWbaRVMaZWvwTmoGbH77l12Hd/V4kt3/UlcHAAAA1oIAb0N8oi4OSjQHQbQWVQLAqjgbFzCzyUrSXpMVqV5GrjwHb6m7rHEf2Rq8YcMMXrZBy3bSvOwzeCtmJj/12BV99uK1lW4DAAAAmIUAb0Oiaommc/m6PKlonjIrUMrHJFTOPwoGnftgZNnOj2EwU6e5SBgEdjUmIU601Bo8vz3LZvD8GrxVSzT/7r/5mP7BXZ9Y6TYAAACAWYab3oB95RNv4SiCMIOXz8Gb10XTTpdylubgrTjoPCxHbN5kpbsumlGk5mvwsost20nzapbBW/VxXT+JddMBA9cBAEA33v7e+/XzH31EX/KMM/rRb/oqHQzJ5+wb/sU3xDdT8fFCnJQbrPif5nXRjCI73WRlUjRZyTN4K45JqP48S3iRWaWlq/JdNPM1eA3m4EnS8WTZDF47TVYmSUInTmzcxavHevr6eNObAQDowHs++bjuf/yqPvDpi7p49XjTm4MNIMDbEF+iGQdNVsIGmvmYhDkZPLMiEPQBjxcHa/CWHZMQNyy5LJVodtSM5FQXzaZNVk6ab9g4TvLSztUDPEcnTmzct/3UR/W9v3jvpjcDANCBcGkO+xz7iRLNDclLNIMAr5TB8yWaM65f7aI5jCKN4yI7Fc7BW6XJyiCyrNxz8eVLXTS7GpOQd9FMA7xaawODyyyzBu9a1kFTWv1x1X0ugS49ef1EZ0Yc3wOAPgoP7FM0tJ/4ht+QapfMOHGnZuCZaeY7M3HpqAV/O8NodgZv6RJN5/JmJnVuo7QGr6sxCZUSzTpr8MLs4zJdNH15ptRCBi92lGhi49IS7k1vBQCgC+E+W1dN77DdCPA2xAdzPl7wgUvINLuLZt5kJft9WC3RdEWmaOkSzcTlgVSdGHEdYxJ8IOwzeHXm4MUrZvDCAK+NDB4ftti0sAkTAKBfwsotDirvJwK8Dcnn4CXzMng2tcmKc07OpX/3pZyDqPxPmSSrj0lIEpd3Xmq6Bq+zEs3scR8M649JSFbO4BXNKFZdWzhhxxpbwDnWZQAovO9TF/QN//w/8rnQE+MkySuw2OfYTwR4G2L5HDwf4GlqBm9anOTPC9fgDaLyyIQ4KXbgxku+uWPn8kxZ0y6aXY5JGDQckxBu16ZLNOMkIYOHjYsdpcIACvd+4Wl95MEnl54Vi+0ynjgdNjhAj/4hwNuQqDLnLsnmu1UvM+1t6d+s4Ry8KFiP5y+TrJzBU54pq7MzuI45eNUxCbXW4IUlmkt00bzaYpOVtIvmSjcBrCwhwAMQCPdFsPsmSVLsv7HPsZcI8DYkS0AFGbzyoHNJkk3P4PnrRFER1JlUun4SNllZYUxC3q2yxm2El+jqA6U6JqHOHLxVu2iWSzRXX4PX1YxAoK4k2f0SzQcuXNUjT93Y9GYAveA/D/apnO+Jayf6Mz/0AX3+0vVNb0rrxnGRwWsStH/mwtW9eg30GQHehvgSzXwNnnOKqmvwJL3nk4/pG3/0w6U3XKlEM7i9MAMYu2AO3gqDzn0gVecDItzGrkoCqmMSxpPF97NqF83LYYlmKxk8PjyxWWmzn01vxWq+82d/W//nv/vEpjcDKxrHiR67fLTpzdh7YT+AffHgpWu69wuXdd9jVza9Ka2bxEmjCixJevzykV7zg+/Tez75eJebhjUhwNuQyCpdNKdk8MykTz12VR/49EU9feN0Fikt0UyvE0XlNXzJinPwfLB2MCgHonOvE1ykqyyVnxfo/6u1Bi9ZLcArlWiuPCaBNXjYvMTtfrOfa8cTXTuZLL4gttrPfeRh/cl//L5aHZHRHf+dveMfC42EXcz7Zpw4HQ4HkuoH7ZePJkqc9FSwv4ndRYC3IT5ZN3cOnorfL1w9zn/2b75bbxqV1uCFAWJ5Dl7zDy8fhORNVhp20exqnVmcuDyoHUZWKzsZbvsmSzTToHu/SmCwnRK3+5nkPjwGSJeunejq8aRWuT26E+cB3v68p/xj7eN38iROdDjyGbx61+nz87GPCPA2ZFDN4E2bgxf8euFKEeBdzH4+f8thUaIplUo80yYr6c/LNFnxO075mIQab/h1NFlxrnjuDgZRvRLNVTN4LZVo+uuSwcOmJW73d+TiZD2NYt72ix/Xe+9rVrI0iRN98DOXOtqifvE7k7v+etx1fjdhnw6ahEtk+sQfTD5osMQmvBzvxX4gwNuQU2vwpmTwwoCvFOBl2bzbzh0WJZpmCq+erJjBS6oZvOA2fvjX79cP/dqnT10nvJeuSjTjoNvoaBjVLNEsfr4xbh7slgadr3CQuVjEvvxtAG0IM/y7Ks2Gd38/P3335/W++y40us77PnVBb/oXH9LnLl7raKv6IyZrsBXcHu7cJ8H+V5/4yiafwav7+Poa8O4rArwNqZZoJk5Tm6x4YYDnf77tloP8dsxUChDDZh51gqAq/3kwmjIo89fvu6D3fer0Dk+ahUx/7uoDM8x0jgb11uCVmqycLDcH76BBqeosE45UY0ska8p+dSltFNP9Y1gm23kt+5wJ1+9iur7uZO+afTwA6V9yO/5ReIrvu9A4g5f40549IXuKAG9DfDAXlmgOKlMSwggvXIOXZ/BuOczX6VXn4IVzrpZpslKUaGaZxtL6Ojc1K5g4aRg1q/luKu2i6QO8qNa6jVKJ5mSJAO94omecHeX3v6w4ZkemKdeDZiDbKAlKuHdVnKzntRF2JK6LoKU+Ste3wz7+O+SPuWfvU7/P55us1A3a+/p81BUnTj/0a58uNTXcZZ0GeGb2WjO7z8zuN7PvmvL3bzezj5vZb5vZr5nZl3W5Pduk6KI5r8lK4WKpRPNE5w6HOjMazB50XirRXCKDl8wu0Uyzg6dv07niMXSVHUhckak8GEQa15nPFzZZWSKDdzSOdcvhUNJqH3z+32HXMyfr9KO/8Vm97p99YNOb0TvLBC3bZl3D2pfJdq5SHr9v+ppFkaR//zuP6vEdGQGxj4PO8wMxPXvM1RLN5mvwutmubfeZC1f1g//hU/rAp5uV5G+rzgI8MxtIeruk10l6paQ3mdkrKxf7qKQ7nHO/T9LPSvr+rrZn2+SljEGAd7rJyvQumheuHuu2c4ely1RLNOOkeLPWCYKq/HYdTAvw4mRqVjBJ0s6WUndfEnFSlIGOBpHGNVpr+8dy02iwVBfNcZy0UqK5j3OGVvXwkzf08JP9G0K7aYnb/ddh4tYzy2+ZYHjfj4Q30dds5yRO9K0/+Zv6mXse2vSm1LKPg8772jWyyOA122/Z94ZHk55VWXWZwftqSfc75x5wzp1IepekN4QXcM691znn994+JOkFHW7PVvHBXLiwuZrBC1W7aN52y4EkzRx0HrYQX6aLZp7Bm9JFc1aDhsQ5DQfl0tO2ldbgDWuuwcs25ubDoY6WaLISJ64YGLpSBm+/j44tY13rrPZNP9bgdb9j5pyTc83HvvjP9WU+e/dNXw98TZL0tbMr8/32skSzp9/Jfr/Il2jWbXrX1/diXUnPDsx1GeA9X1J46Orh7LxZ/idJv9zh9myVvEQzaE1czeCF5YTVLprn8wyevz3NLNEcL/FirWbwwp3BcZxMfQM4SYOoWdempspNVqJaj80/xzcfDpYakzBJXOMjYVNvJ+7n0cIuxc7t1aL/denDDLl1PIaifLBpiWZ6SonmYn2dv5ZX0OzIayA/2LxHn7dJT4NaH+AVY67qXW8fg/xQ3wLcrWiyYmZ/UdIdkn5gxt/fbGb3mNk9Fy70pDY2e+aLD5jTXTR9A5Hn3XpGT1w/yd+0F64c67Zb0gAvCsYklAadtzQHbzTw4xzKf5veZMXlJZpdjUkI1+A1LdE8ezAslWhePZ7o5z/68OLrBxm8Ntbg7euH5zISMnitc9lnw67vUK9jDt6ybcMp0ayvKAvb8Ia0bNd2Fots1m5sbxvinnaN9Ptn/sB00y6ae/QSKOnbwaYuA7xHJL0w+P0F2XklZvanJP2vkl7vnDuu/l2SnHPvdM7d4Zy74/z5851s7LpVm6wkyZQumpkvP3+znJOeuHai40msy0eTPMDzMZ1ZOUAszcFbYqGKf6OPpqw9C0cwhFwQfHVVmZQkLn/MtcckZNt6y+FAceLy6/zKvV/U3/7p/6xHnrox9/qTOClKNFmDt1Z9mNe2bfzTuevP6zrKTJct2fE7jGTwFuvL67GqOMC6G49rHwed93WwdzWDVzeA7VuJYlPFeuANb0hLugzw7pb0MjN7iZkdSHqjpDvDC5jZH5D0DqXB3eMdbsvWKQK89PdpXTS9Lz9/s6Q0c3fp6okkBQGe5aeDaonmCiUieYlmnrkqXvGT2E3tzFleg7emEs06g859k5WDtBPmcZb1O6mczhInLmg2s9x2S8zBWwZDkNvXly/xdXQCXTarscr6533Tt6PmXhHk78ZrYC8Hnffks7CqOiahbgXCvlce+Ifdl/fAsKsbds5NzOwtkt4taSDpx5xz95rZ2yTd45y7U2lJ5i2S/t8sUPm8c+71XW3TNhlUSjTDwKXqJbfdIintnukvX22yElmRzfO3VxyNWKVE83RgkwYq00o0lQeZXZVoxkEzmnQO3uL78Y/Flyv4na7iw2z+8zNuqckKGbzmwjbWkWakuNGIf/3t+ndYWqLZ8X0sm8HzTVZ4ry/U1y6a8Y69BvoaaM/T17JUf1ChqDyqd71976LZt9dDZwGeJDnn7pJ0V+W8twY//6ku73+bWYMM3ktuOytJunT1JA+cbjvVZMVK15+EGbwaQdAHP3NJn714TX/+a16kj37+SV0+mkgq1uCFgc2sI5KJc/m4hq7WTSVJ8ZgP6mbwKgGeX9tYt4wqTlzjI2HT0EWzuTh4f4wGm92WvvAv4V1f27iOOXjLrg/ra2agC3098JV/v+xIFrfYF9nsdqxTXx/zuDImoe6B6b4FOE317XO70wAPsxVdNIMMxYwA7/wtZyRJx5NYF6+47Lxqk5XqHDyX78jV+YJ5192f1wc/c0l//mtepL/y/3xEL35OWhY6bQ1ePCtgdH7gerdjEgZ5iWbNNXiuXK7gyxcm+Rfw7I11WRlYG01Wwkxhksz+90Zh348odqEvZTjrWJ+Zl+w0vB+6aNZXdCrd7Ha0bdcyePv4WdvXx+z3aZr2DuhrwFtX3w42bUUXzX3k9+39G88F5Y1Vt51LyzGPx4munaSZtVsO09g8n4MnKw1GDwOfOl8wx+NEV4/T237q+lhPXE/X+vkAL7yNcZLM7KKZBnjW2gfmz37kYX3dP3n/qfvw21ari2a2rWdGvkSz/Cae92aulneu8rjCQHLXsyfr0ocP3KvHEz30xPYMa+/LUdrEdb82c9kumnmJ5r7uKTXQ11b1u/bZtY+Dzvu6xnuc+Dl4zQ5M97XpTF19K1MmwNuQJk1WnnHTSFJaWugbhBxmwUrYRTPswhmWZdYp0TyaxLp+Euv6yUQncaKrWYnmtLVn8wadWzaPr60PzM9cuKr7HrsSLAAvuoWOhvXW4Pn3qs/g+Q+/OjsWPpBtJ4Pnpv6M2Yov4A1vyAre8b7P6M+944Ob3oyc60EGL1ky8Gp8P0s+V3mTlR1+jtdl1wKhuvxn1q500exroD3Psgdwtl21yUrdh9fX92JdfeuiSYnmhvggJdyBmNVkxXdwPB4np86zvESzvAavlMGr8Wr1t/3Fp48kKc/mFd0jizf8JHGyGU1WrOUSzXANzMCy0sZgDV6dDmV5Fu5UBq/897nXbWUOXnHdvhwh6lofSmguXj3R0zfGm96MXBy8p5xzpcz/roiDAz6d3s+Sr79931Fqom9Hzb2iRHM39haTNb2ntonr6WP2+3x+n6duALv3Gbwe7G+EyOBtSF6iGbygBjP+NYaDSIPIdBLHOoljDSPT0Ad4/vaiItgbRlZq/V/nKPLRJB0AXg3wfIlmGIg6N708yrk0+Ioia23HprqjlJS6aFqjQec+WPXBr18TN+8Ia17LPmihRDNcg9ePz4/O9eGo8vEk3qpMTrgpW7RZjayrnGzZQM3lO/c7+gSvUb6T3bPnapU5tJuwlyWaPX3MfjRW40HnPX0v1kWTFbSiSYmmlL5RfZbNlwtKQYmmLC/RPDMalDJ4dRqR+Nv+Qhbgeb6LZlFyVNxWtXV94pSvwWtrTEJ1dlyY6RwOolrlp3kXzVF5PWGdDJ5/vPmRsFXm4IVr8HryAdI1/5Tt8hfO8STZqu0Pv+wXfe5sq3UF/v7mm/7z5dmbvtT6dKivZXK7Nioj3BfZF3mX5p699vyB7+ZdNLPTnj0fddXZJ9wlBHgbUgR4RQAzq0RTSoO648qbNrwdP55ASpuJhIFPnSOIPoP36FM3Tt2vNL3jZLV1vc/gDSJrr0Sz8iXpnEpNVk7iZGGZWbWL5rgyB29eCY1/o7eRwQs/NLZph3+brWutVZeOx/FWbX/pdbhF29XEukogl23CQBfN+vxztSOVjLXtWpnuPpbn9TV7nB+Y9uOd6jZZ6dkatKb69h6gRHND/EFzNyUzNc3hMNLJJNHxJM7ftFJ5Dp7NyODVWQPgM3iPXi5n8PLAZkrTgOrOS+LSDYqsvR3y6pdk7MI1eDZ1O2bdhg+Mx5U5ePPezEWTlWYflPNuS9rtgGWddm0naZrjSSLn1FpWe1Xh631Xv8j8R1rXm79qF81dft2uSx/KsKfx//Z1Kmi2Qd/WH9XR18c8PjUmod71/HtwW76r1o0AD63wwVy4A1stlXr1K87rW1/1UknpkZjjSayTSZKXC0ppaWZ6e2EGb5CvwUtnxU1/sV45GusnPvg5Oed0PCODNxqWF+mWOkFWbjfJgi9rsUSzWiNfXoNXDthm8W/WM6PyHLyijGr2tvr7r5aqLoMMXnN96KJ5NE7fW9uysx8+l9uyTU2ta5bfsmtS6KJZX992qjz/cHblPVYclNjwhqxR3NPHnDdZabgGrw8HVFdRPP4Nb0hLKNHckOoaPL9+LfTjf+mr858Phr4csciqSeGYBMuj9TOj9LJS1mlyxqv1PZ98XG/9t/fqj7z0OUUGr7IGzwdTeQZvTmbQl08OrP0mK/lavKRoJpMHeBMnHcy7jfT0MC839U1WFn+Y+eBxNIiy7qDtZPD29POzsV6UaGYHWyaJU5B835hSBm9Hv8jWtW5r2fuJp3xeYrq+Nrrwr5nxjjwu/1nQt0B7nj50aZ7G72v4aq+6jy+vKOvZ81FX3zK6ZPA2JMqe+bCUZ1YXTalosnJczeBZkcHzP58ZFiWah6PBzKPIfsfz2nFcdNGslGgOzDSMLFivFmTwTpVoZl00WxyTUG2hHXYb9dnFk0UZvFMlmuXAbt5Rdn+ZQZSOofC/H41jfeiBS80eS6VBDRZrkql57PKRPnvxWteb1Jg/eLItXxrha29XX4frWjsTBweWmujDrMF16Ws5a3EAcTeC/L6Oq5inr41lqiWadY8z9fVgS119+ywiwNsQn63Lyzicy2fjTeMzeKfW4PlTszzwSdfgFUHNrADGlyZeO57kl3/qenle1yAyRZFNbbJyeg1e2uwkiqy1L4npa/DSR+3X4C0q0cybrFRLNGtk8PxjHA3S7qD+tn7ptx/Vm/7Fh3Tx6nHtx1LK4PXkA6RrTY6wft8vf1Lf+q8+0vUmcQmUUwAAIABJREFUNeYPnmzLl0ZYPr0t29TUunZGiy6aDTN4O9ZBcZP6dtTcyxuE7ciYhL7t3NbR19fe0iWaQUXZPupbF00CvA3J1+AFR6IHC5qsHI+TdA1eqYtmcRquwQsbi6Sz606/YH1g9OT12UOYfcll0WQlyEIlTj999+f10BPXJfkSzfQ6bQUwYRDmnDvVRVNa/AVabbLiH0OdL7Qig5fOIvSP6+rRWM5JV44mjR/LovtEoUkG78rRJH8tbpM8g7clB/LD4yG7umNTHdbe2f0sudPbtx2FLhWlgZvdjrbt2jrMJHhP7Yu+rv/0ZcGjQbMxCX1YErGKvmWxCfA2xMdy5dLDeRm8gY7jrERzOK1E0/KfbzooMnzFaIDZAd4T109m3m9Rmpj+Hu6wHI1j/Z2f+5h+/qOPSEoDPJO1WqIZBmH+NqsB3qISTb8D6J+3k2BNlL/tWfxlhpFfW1g+/8ZJXPuxlEZM9OQDpGtNdpQT53TtJNbV4/pB9zr4BkZ1utmuQx8ONCSlILW7+1m6i+aOdVDcpL5mjpIa3y/bZNmRILusr6+9SZyk+yxRudfDIns/6HzH3rOLEOBtiH/jxXGamZrWZCWUZvBiHY+T0qDzKG+yojwDeCb4ezHc+/SORl6WeW1egKdSc5EwULyRdQf0AVPinKJIioI1e6sKO176bcjX4NXsollk8LISzUqd+fw1eEl2n2npqf/dB5X+Oaj1WIJ/g2lZhz/+A+/VT33487Vvbx/4f6M6Lyf/7/jFSqOgTTsal+cublofxiSU1hF2+GW8dBfNnu44dqFvR829OnNWt4nfzG35nFqHuKfZ47Shl+X7h3X/Tff9c6tvJbsEeBsyjEzPvvlAjzx1I/9wmZ/Bmz4Hz6/CS9fgpQ1RhoPidqqNRUKTGRm8MM6MzDQcRFPXq/ns1TgoeYwsXasWBjBH41ifv7Rc6Vx4hM3fd9FFs+4avPQ0D3b9oPMai+D98+aPhlVHKxw1CvDCndLy35xzevDSdT34xPY1CdmkJjOy/E7445e3J8ALR5Cssp935Wh2GXVTfeqiKXX7ZRyWgjZR5+ARUn07au4VnVR343H1tVxxnr5mrMZxotEgkmXzkZuWaPbs6aitbxldArwNMTO94rnn9IkvXil1apzlcBjpeHJ6DV446Dwyy9r5hwGebywyLYOXnldtrHLbLYf5z4MovV2/ozIO9giPsszdeFJ8KFi+Zq+4vZ/88Of12n/2/jzT10R5DV6xTVKDOXizumjWaIQQ/ttEQYmmv88mJZrh3MDTHUiz253044OlLU2OKPoj5dVOsJs0jovS4mWPjH/u4jX9/u/9Fd37hadb2abwqdzVo/XrykIu+4Xftx2FLhUZvA1vSMuSGt8v22QfOyiua9zKuk1il+8fDax+07sio9mv56Ouvs2CJMDboFc875w+/diVfMd0fonmQMeT02MSwiYrkaVZrTBQLBqLnH7D+oW4T2YZvDPZ7d5+rgjwoqw7Z5wkunw0PrUGTyqCHef8oPPyB+bFq8e6fhLr0rX6HSe9cJ2cv03/8PI1eAuCoryLZl6i6ZteLN4Jy9fgDSINonBtTXrarERz9k5psSC/J58sLWnSRdM/ddsU4PnsnbT8jtOjTx8pcdLjl5u/f6bpwxq8dT2GZWeD+WM5rMFbLH+Od/S1OEu+XntHXgP7eFCir495kqRr8CSVun8vQolmetpl4651IsDboN/1Jed0/STW5y6m5YuL5uCdTGIdTxIdDMIxCVmJptIs08GwksEbzc5yjSflLpo+cxcGeIOsucjP3POwvvb73lPKWJ0K8KSpJZr+OhevzF7rN0tYNpLkAV42JmFYs0TTZ/BG5QxenSYrvnwzb7KSl2g2X4M3b6c0zgPH3dgZWJcmXzj+sm0FQm04DrLWyx7J9+s928oE9GINXniwpMO3zLI7PH0tO+xCkzLsXbJzXTRd+XQfNFnjvUtOJkUGL4rqPz4X7G/to759Fg03vQH77BXPu1WS9PFHL0uq0WRlkihxrjLoXPl1v/4PPF9f+bxzeujJG8H1yrPfQv6L58msycpzbjnUw0/e0O3nzuSX8c1FpLQN/eVgLZAP8PwOaOJcHmhOy/Q1mRnnhV+S/sO42kWzaYmmfy7qlND4y/rnododb9k1eNUPUJ+5m7ZWcp81Obq/jU1WwtfHsjv7vrS5rYHJyZwDDbsifPt0WqK55KL7Xdu536Q680h30a5lh/rWYKKOvmasJkmS92JIu3/XLdHcv9dAqG+fRWTwNujlz71FZtLHv7A4wDvIArxx7Cpz8IomK3/oy5+jb/7al5RKNA8Gs7to+sDMl2iev+VAkvTcW8slmk/fKIK660EGL2+y4gOmJN2O6pgEn+W6sEKAlyThmIT0tHYXzezDahRFMiueiyaDzoeDcpMVX97aaA1e8G8wq0SzrQzef/rsE3r06RuLL7jlmqyR8Dvjj13ZngAvzOAt+6U5bjmDt64OlF0qPYY1NFlpPAevpzuOXfDPVV/KorxdHXS+Tzv3+Rienj3mSexKJZr1B53v9+dW3wJcArwNOnsw1Jc9+2zePGFRk5Xi56CLZrAGz5teojklg5d9uvlh3c+5OQ3sbquUaIbDvEslmn6eXJDBiywdkxC+QfISzRUCvLCL5ukmK/PfjHnmL0qb0JzkXTTL9zHN6Tl4y5dozuui2XbHtb/2kx/Rj3zgs63c1iY1a7KSBXi9zeC189pYV/arS+USzQ4DvCUbgPhghZLrxfxLsG87lbu2rrooK97whqxRXpLYs9ee76IpqVR5tEhesbBHr4HQrmXdFyHA27CXP/ec7nvsiiTlpZDTHJQCvKBEMzsNg7pwLV+1LDFUDYxuO5dm8G49M9JNozSIHFSyitdOTgd7RZOVYg1euOPoA8Fl1uDFwZe//zD2YxIOGmTw8qAwsmK2XrJ4x7mYgxeVAtdlmqzUWYO3aGh7XUfjpFH56LZq1mQlvczjV4635gs7zOBtS4DXhyYryZoyeMvuAPat1KdLRZZ+wxvSsiQ4OLAtn0fz9HUe4Tx9y9h4fg6epFMVVfP4y3XxmboT74GeHeQgwNuw8+cO8zEF1WAqFGbtDqaWaBaXDW+n2jkyVA2MfAbvlsOhbjpIrxdF0l9/9Uv1+1/4TEnVDJ5fg1d8SEZ+MHpw00crZfCKIdH+8yEP1mo3WSmek+EgKubgZbdXZw1eNYOXr8FrUKIZBtSn1+CVM4OrmiRJL3Yuiwze4sv61/gkcbp0rfnBhC4cj1sI8Dos0dzVl0h5Dl6X95OdLtlFkzV4i/W1PX/4mbULr4O8ycoObGtb8u+Xnj3kcZxoGGVjEqIGXTQ7Cng/+JlL+j3f8249db3d7+XHrxzpdx5pZ3yQ1L+DHAR4G/asswf5z/O6aM7M4Jk/DTN4py87dUxCZa/5S5+ZNld51s0HRQYvMn3n132lvu3VXyFJunZcBDR+59V343RKu3pWM3g3Vmqy4k9Pj0nwH2AnCz6dXRZ4SukYiXFlh2Je8wr/gTccpE1WTs3Ba5TBq7MGr70szS7sVCzS5AsncenBCUm6//GrnW5XXUfBmIRlj4q2X6I5+0DDrgh3QtdRorlsF81daZG/SX1d+7VrzYzy78Oe/TvM4x9q34LadA5e0Z+hdolmRyWrDz15XddPYj3R8oHXd7zvAX3Lv7yntdvrW/djArwNe9bNRYC3qItm/vMoHJPgr1tctlSiOWdMQrVs84+//Hb9+Dd/lf7gi56psz6Dl48kSG/nxnh2iWbinMxOL+pdJcAL33DVLpp5ieaCAepx4ooMXhRk8Gp0uvN/G0TpPMDqwvkb4/o7cJM5X/j5IPnWMniuFx9SxZqAxY9lkiT6Iy99js6MIv3y7zza9abVEmbwlv3SbD2Dl4Q/7+ZrZF2NYlbtormrz+869a01uRc+nvEOLGrqaud+m/W3RDPI4DUadN7Ne7Grz8PrJxNdD5YNrSrPYvfk9UCAt2HPOjvKf57fZGUQ/Hy6RDMMDsO1fPPGJIwrb7Yzo0iv/srbZWYzA7zrU0o0q2vwBpGVu2jmJZrNj96EHS+rc/Bql2g6l19nOAjX4C3+QvPB4DCKyiWa2WmzLpqzMydtdtFMEifndqMsaJEmawKSRDp3ZqQ/8ZW3666PfXErdq7DQeerrsFra8erNAdvC56jZcx7L3VxP02fpnxe5o4+v+vkP/J29bU4S2mt6w7UAC77Wt9lfe0aOY6LNXiDoPJokfz7tuXjEfkSlJaf50nc7oHsvh2YI8DbsDCDNy/AC0s0w5+Vl2gWZ5XX4M0ekxBmvg6HUanM06/B89vk7zMs0fTBTWkNnqXbMm0O3pPXTxqXLIVHVE6twWswBy8KruODszo7YWEGr9RkZbLaHLyZXTRb+GApBrhv/1HjRZp10Uw0jEx/9vd9qS5ePdaHP3up681bqI01eG2PSSiVju3okcp1dQJddgfQ9XTHsQtF57oNb0jLkp3L4KWnu/qZsIxlM/TbbpIkeYWTWf0RJMXg95YzeJWqqdZut+WlKH0rFyfA27BwDV7tEs0FGbwwUJw7JiEpB3ihswfD7HbT3/2HRViiOX1MQroGL/yAuDGOddNoIOfUuAY7DHz8z/6h+jkvi9bglbpoDiwPzuocrfF/Gw0qYxKSJdbgBdt5ukQzvb2TBeWmte4nX/+z+x9STb6A4yTNXr/6FbdrNDB94NMXu968hY5bXYPX0qDzYDN2NQCJ5xwsaZNbMtvZ9lzLPuuqLGzTdq1b7T6WaPatLb43qWbwNjwHr7MMXstLUYqqrtZucqMI8DasbonmwYw5eP4aYWxYmoM3p0QzDIzOBOv6pDSDl2bjLLud0xm8oyljEqxSoumc09E41guedZOk5sPOwyDM72z558nMdBB0xZx9G8VzMoyiPJhKlsjgVZuhNCnRnCRFoFk9QtZuBq+bo2Wb0KSLZpxl8G46GOjswbDRv01XjlrI4PlRC9100dzN18i61uCV1is2eK7C8S6Yr6uswaaF//a7cLCtr+vR5snLg3v2kMdxoqGfg2dW+/F11Win2I9rN3LyGby2Pjv6th6YAG/DlmqyMqWL5qwMnu+GeRKf3tkNAyOf6fPOjgal28mbrExdg+e/oJU1WSne0CdxosQpD/CarsMLv3SKLprFdo0GVqtE0zeeSS9fPkpVJ4Pn1+AVc/Cal2jGQdlE9QOkzSP+bQaLm+Sca9TlLE5cqXy3rZmCqwgzeMvuOPnH0dY6nnJ3v1Zucu1KXTTXUKIpNQvWkp68B9ehr1mU8HW5C6+DvpbKztO3kjxvHDuNomAOXs3XXz4moKsMXssHOoqD9e3cHmvw0Kpzh8O81LB+Bm/KHLzgsmGTlVvOpKWW16dkM8Jg4sywnME7ezAoBVJ5k5WwRHPs1+CFJZrlLppHJ+nfXvCss5KkS00zeMGXjj/4UwrwhtHC0QKxc+U5eJUM17w3s/9giswvVi5/UDUp0ZwkReviWYPO2/gAnPTkQ6pU4lSrRHN6Ke4mhYPOl/23zUuKWz5KWf15l6wrCxnu6DS5m77tKHQp7tlRc6/crXbzn0WLhOvd90VfDy5MKhm8uo+veA20uz1dfR5Wl8ysfnvpaV/eA8NNb8C+MzM98+yBLl49njsHr9xFMyjR9Bm8IKgLm6z4uWDTytXCHc5qBu9VX3l76aijzzxdD5usjE+PSfBr8PwbxF/GZyqPG+50h6n9ootm8fc6mZqwycowapbBm8Rp2Z9ZVqKZXXS5OXhOB8OBpMnMQedtZJ3a/tDblKbZk/Jay2grjpqHGd6VM3htlWiG2a8teI6Wsa71TU0PMnhFpn83n9918h9Tu/panGXXMnj7WaLpH/OGN6Rl4+Bg8iCqPyahqzlwXa7Bk9rb3r4F/AR4W+BZZ0e6ePW41MWyqjwHLwzG/Hq04pwwUPQB3rQM3kmc6OaDga6dxKcyeK9+xe169StuD+4z/fu1kzCDVx50njhlgVDxgekDoFuzTGLTLppFEFa8+Uqlo4No8Ry8yo6/n5tSdNGcP+jcX3cQlDr4rmiN1uDFLv93nNlFs4UAry8ZvPCfpV6TleLfajiwLSnRDNfgLXcbJy2vwQufyl3NmqwrC5k0PMhQvd4uZG42LS8L282X4kxhwLrta/B2bSh7W/raWCY9MO27aDafg9f2Z2rcUV+Atpej9K3yghLNLeA7aQ6WWIPnY53SHLzKmISDQTQ1wJvETs+4KW3yUs3gVfkMXtg0omiykr4Z3JQSTR8A3XomvZ9FHS+rwkW//k0XPr5hjTV4pUHnA8s/DOocrUrLKrOBoVNKNI8nSe0vh0mS5P92szJ4bRzx92u1duGo8TyNM3jBv3Od5jvrUBqTsOSX5rjtDN6aGpR0KS4F/93dz7IzA/uyDnYd+nbU3It3KIPXh7LtZfS1g2u5i2b9z8id66Lpq7FaOoBSHGzqx+uBAG8LPOvmNPipP+h8SolmcFX/xpbS0s2bDgZ51io0jhOdywKvagavahTcpv85b7KSJHlDjLxEMymXaJ7LMnhNm4jk2aisWYt/TMW2LF6D55xKc/CqGZF5HzphViia0mRFKp6HRdISzSzAq9xnnhlso0SzJztM5WHW8y/rsjmJYaZ2G8rjwtfGskeJ8y6abX2J1WhQcjSOdfX49GfGtlhXxmHZLpp+k3b9Pdi1UiOlnuxUeeUM3uYPNs1TWtO63ZvaqnzNWc/ep+MkKQ5MN1iD5zrKpsf5spj2u2hKbVa39GPfySPA2wI+gzevi+asQefmSzQ1PYM3MNPZg8HMJiu33pQGXosyeGaW3+9BPmDcZ+7SN0TinMxUGpPg1yDdmmUKG5do5gvwix2ApmvwwszOqHEGLyk1wQnHJPjtqFumGWYDqzuLkzYDvKTdgGBTSp0SF3zg+n+XsERzG2aQHY+TvJPtsl9CJ3kJdLtfYtLsstEfePd9+os/8uFW7q8Ly2bW1nU/fZpF2aVdmxXXxE5l8JY8kLHrejvoPHb5fss2lGh2twavm/XpW/52rY0Abws805do1uiiOYysdLloSgav9PcsgzctCBkHJZqLMniSdJgFJ6NhpOqmjuM0gxKZyYIxCT7A82sBm5ZoxkEw5j+Mw1LWgzolmlngKWVz8HzJW42jNZM4yOAFA0MnQfazbqOVeRm8OGj3u+qH1aQnX1qlUsIFj8X/vZzB23yAdzSJdfYgfW8tG4j4AxjtdQpbHLQ8dvlIj18+auX+utDktbHS/TTIIk+73q43OupauRvqBjekA+Uumtv94NZ1wGTb9LGDq3PpbLhhsLSkdoDXUeUBXTQ3gwBvCzw7L9GcfZlBZBpGVlp/JxWDyK2StQuvd3ZGieYkSfK1cYsyeFIYZEb5Al4vnXfnsjl4lmcJfPBz9mBQa2ZdVZiC9x/C4WMdDaKFR8mTSvONooum8tueZZIUR8IGQenpOHZ52el777ugf/87jy58LJPEFXPwKvcZbsOqgclkyhq8GydxngnaFU0aXFQzeOG8w006Hic6e5gGeNuyBi+8mVnbNImdxlu8o5csGXg1tWwXzbDUaZ92mJsKn9K+PU/h59c2HGyaZ11jR7ZNUaK52e1ok//eL8/Bq3fdPKPZ1Rq81ufgtRs49m09MAHeFnhmjRJNKW2Y4rtZVpWarFRGJpwdDU+VaDrnNI5dXjp5WCODNyuLKKVfYH4NXjoQPD3fZw7PjAZZMFb/k7Q6zNh/7zQu0Qy7aEbFHLw6b+a4ciQsdi597pIig/cP7vqE/tGvfCq/zkf/f/beO2C2qy4Xfnad8tbTa5KTRnoghCIJRaQKiIAFPwvq1c97VfTa7vUWL5erGOVDbCBCPhS9F0VQlBpEICQhhRRSSHJykpzez/uet5cpu90/1vqttXab2TOzZ96Zyfz+Oeedsmf3vZ71PL/nOb6Azzx4InFZgsEL4u9RdSpjSJot+8m/uh+33PZUR8vtdYVcNDMCPFNh8Pqh76Xmehiz2URAuw+Nes49eFkMFVzf7+uHXK+iHoI2mY1eMYyDXu0C6EGoQZKfhnpa+3tVc61uxQJsZNFzwjKVHLwWJZp5X4vd2s/dctEclsmmEcDrg7qAh4AT0EsrmztiqiWCzlWJZgjsgUk0IzJCuiAmM7po0u8DjAUzEwCeCDrX5Q2CJJol2whl0GUpN/KAjLI0AAWdN+/Bo/1kGppkuIT0LRuDx8xj2PKCQEY/rNe9UN7Z399/HLd8JQ6mXN9XAF5UoqnM9nbItEm9u2R+vntyEWeWKh0tt9fVkkSTb7OuSYDXqhy4G1V1fJS4RLNTgNfLHDzXD/qadVAPbVdNVtp0HFV33agPL728NgH0IFTont7n54C664ftODSqYcz+owin8Lgl2/aJ2IiB6cHrEoM3JOfDCOD1QX3PJZtx+2++ChdvHWv4uYJpxIBYkoumEWXwEkxWaNBRtg28/LKtuOGC6abrSeDS1DUYRgTguWEXzahEs2gZsDOAMbWiQbH0t9ZiD54fycGTwez8/YYMni9z8HR2I6GbCjF4QDjvbLXmJpraeF5GiWaHehExC8eP8amFChwvGLiBZicmK5ahDR2Dl9dDJyTRTAN4fX6+hBmHLgK8tl001XvXxp+H/VpRlcYwVbsZihtRWZx1h7G6BWg2sgSD10YPnswFzHedZA5el1w0c3aY7oOhQy41Anh9UJqm4ZJt400/V7D0eA+esgyqkEQzxWSFZI2WoeOTP/9SvPHaXU1/n9gn6gdUy/EZgweRg8der9TZ7xRN1rfXCsCLMg1JQeeWocNxmw/+VRdN2YOXPKv0lg99Cx//1mH2nmKyQhJN2gZyIAUQYvBWay7qrh/bVleRaEYHi3nO9tKAkrbr8PnV0N+DUq3It+IArz9MVmquZPDaHUTUE3oqO6nQwDNlkY7n9zUwUc+HoIuDM7/N3xkked5GVrgfdOPWoxsVlt3377UEDBYYzbOGrecKkP2eFJfFzOGyfbdbEs3Bc9EcjvNhBPAGqGxDj/XK6QoNT2Uosk1NSzZZcQXAa9z3F/p9Dk4sQ2/cg6dL9qTieLANHaahwzI7kGgGgZhVCmf+NR/I+z6TjdLnxU1B3NzD3z88u4Yj59fEOogbJZc60DZMpjB4azw/LMaa+pLBi7toKoOBDoFJtAdPbkt/DzKiFZbHZfusMNPR+yQHz/Ex1rFEk51HeYW5ZmFGPX6edxM8dVJhANV/v+P7gbi3DtrESi+rV72UG1EhFrcP7kWN6rkedD5Mpx6NhyydevCyT04NnIummPzM5yEwbIB/BPAGqBoxeCroITBDQK9sx01WaPAbdcNsVLahMnjh7zluIHrwbFMan1QdD0VLAsOWJJopDJ4ectHUWjRZ0cTANY3Bq7t+yInS0MMmKwTAyEWTvkPru1Zj+zoJVJPEtpHJSscumpHtIoDXD4CnlWpFvkXHi46zbfZHDl7d9WQPXtsMXvNe0VYqS2+j06UHcl7VK4lmu8yGH8jJnH4f3G9kDTNzNEgsbrtxIINeWdo0Bq3ofkMT0+0EnUf/3/E6DUwPHv93SM6HEcAboJoqWSK3jkprYLJC7F7JMlBz/UQQ0QqDRw6eZgKDRzEJuqbBNnTRN1R15ADXalWimdKDp4d68Fo1WZE9cEmzSj7vsZPW9L5iN8xulDTgVhk8QLJ4q5zBI6CnroeVKSahs5uLF5GgCgavDwBPK9WKwYWQ72oqg7fx2+v6gWDd22XgpKQ4p1nKDAPPLAZEG1m9MoVot0fMC6Qce9CY817WMPd++YE0uujX64gqFHTe5+uaZw2bqQYg7zc01tGUlplm1a1Jie7n4OUr0RyW88Fs/pFR9Uv94TuuD/XXAarJinxdnbkBIIKWK44nAsclwGudwTOTevA8H37AbiaWwW4oruej4ngocWBomVpLs9nRmw39GevBa7LMIAjn4LH1Tb6REXgTrEmkB8/3pfnEpjEbr796BwDg3/afQ42zNatCohlh8LjcU9OSXDTlEzY/Bo8t5/DsWuj1QalWBn+0bXR8s+Qj9qJcLxCsbTsPDXUiIr8ePOX/DSSaADsXiynRLBtZvcrtatdF0/elpP25NGButYZZGujznmu37vU9yB/m49CougU8NrJobGOJcUv2SbBo33teAGHQcvCG5RoYMXgDVBdsLmPPdCn0WqLJiiYBCSABngo4RBhmCwCvoJisqKAHkE5/GuTApu75qNQ9MUDMklmnVnSAL63w5WeyyD69QDJ4pEuvuZJdSwJ4ari0AMwk0eQP64Kp49Z3vQivumIbANZvFQRBag+e57PIhSTJRK4MnnLTqzoeTvN4hMGTaKr/b8LgiXODH2ezuXS32xUEjO0tGDoD9W08hNRw+rweOqpkOVWiqUxw9GNlYSHzKPUUap/B68992A81zMyRFwTimdmv1xHVczbonJ9zw7TJUqIpc/BaddEE8nXS7LqLZs4MXp/Px2SuEcAb8KIBbVJMAr1W5jbtqpMmDRzNtkxWJMArc/BGy9M1TTzUag5j8ATA68BFMyTRDOXgNe+18nyEcvAABsbUZVNFQ6UdpQePcvDqLoFjtqwil+DVXA811xfLUwF1EASin0/X48Gj6oC1Uyml6qJ5fH6dm98MuEQzK4OnBNpvtERThK8bOgP1bYwiVICXZ9A5Xb9pz8Vu9UzkVb0akLYt0fRHPXhZKnyNb+CKdKE8f3Bkur2aMNnoCoIA+08vi7+F0doQITyRgxdy0Wydwcvzvqp6GuRZbs7AcdgkmiOAN+BFxJ0K09IZPAnw6EKLBqc3KmmyooubB/XXEVtCJiv0WjUniWaayYqdQaLp+wFoM2lWS4018BIAXl3pwTMVqYPK4JHRDEnwqo4v2Dsg3INHP0EMXvT+kSeDR/spCIDligMAGC+YHd9cXc/H+760HzMr1Y6Wk7XCBgyNPyvYXSUmwQ82drCiykZ1XWtr/6ssZG4yFJ+St3PPAAAgAElEQVTFhugNWEX5QO7PgWl4QNq932nFyZWKzAlsPvHTr/uwH0o9p/vVsbXd8geIxQ3LtjduPbpdDx5dwJv+/Ft45twKALndwwRqRQ4emcO1EHSuXoJ5gpyu9+DlOPmpLnfQq6sAT9O0N2qa9rSmaQc1TfsvCe+/UtO0hzVNczVN++FursuwlnDRjGTfqf+WEgBeNCslSwkGT9cEqzXGe/pI8qjrmgCCdZf34NlSotmuyYrny5gEQwF4lqHD8wMsrNVTGSpVkmbz7a2FmJF4/5uT1IPHpZVC4873h8rgqaAuLIllyyN5a/QGEjLA6fAJq97sKGi+ZBsdM1pH59bx8buP4FvPnO9oOVmrnaBzAuOy13LjRisEzmzO4LUl0VTWP88ePHEepjzEBQvcp7RKr8w52nF5pM+NevCa17C7aNKzMK+Ik27VMJvdqLWwXgcALK6zic9WnjGDUm5kbKdr2Z1RQ8qIHPfHoLhoDltsS9cAnqZpBoC/APD9AK4G8P9omnZ15GPHAfwMgL/v1noMe0kXzTjA04XJSlyi2ZbJSkLQObFzNS551BQGr+ZSDx6Zs+gitDlLRWUjdPNRHUNp/W9+/+342/uOpS5HSDQjPXiGHnaYIgZPDUO3FKmD+hlqYlYZvNUUBk8FIJrWxEXT7QyUqMsmKWrZNjserEfBb7erpaDziHxXyOM2ksHz5DFnoL71ZagSzTwfYromcx2TSkiU+1TW288STVq3Qh+cg/1eww3wJIvr9Pm2hY7DEAM8oUwghc4Q9h7SuUZjIz1hQjmtBslF0/cDwTh2w4BsGK6DbjJ4LwFwMAiCw0EQ1AH8A4AfVD8QBMHRIAi+C6A/RxEDUNJFU76WyWSFWKg2GDxT6cEbKxB7Fe/Bq7s+qo504bNNraU+sGjQOUl4wi6aGt82DzPLydJBlcGL9uBZhhaSUNXcMIjxlB48Yg6rHBwKBo9vX9XxsKbsY3V/E2Ak5iT6MAn14HUqpVS+Lxg8y+hYKkb7pFfmJa3MJvoKgAbkedEpWO6k5EyqnnjMs1Q3AJ4fBNBTmGSqfu/B65nJivrAz/g7dJnRxE+/sqD9UMPMHLFeV27O1ee6R79LzE2/VfQZ1qs8zV6W68mxDcDGZFnlz+EJrRzXqQsAz+3CM2CQsiuzVDcB3h4AJ5S/T/LXRpVjaVykqSGdwSspMQlUQqLZRtC5qeuSwePsoOqiWQiZjnji71azyaIXG31Vj0g0qdKAh8d7jtTPE4Nnc4lndBmO0hRM20rsELGV9LqUaPpYraoAL5nBa+6imSeDp0o082Hw6j0CTSGHvYwmKyIHjx/nvpFotjCLqhatf7vfTyo/YNdDI+MXt8dsbasVcgLt4oO4EwZPssj9uQ/7oUJOuYM/ngoVPXdMvbXe840ouoa0FuR8g1hRd+Bh7D10hHJETkxnBa+h/ZFrD17+uapRE748atjiQgbCZEXTtF/QNO0hTdMemp2d3ejV6avSExg8I4XBUyWDoo+sDYkmSc4A6aIpevA0TZqsuD5qji8kmlky69SKyneki6b8jLr+aYNR3w8EODMjIK1gGYkumvJB4Md6Gmlb6belRNMLSTTDpjZ8oG4wF814Dp4K8PJj8Goc4JVto2MXTXIP3QgGr2nQOQG8iERzI6VR0q5ag661Z7JCjHI5BwaWyvOZpFvX0yWatN/6dWDqB3KGupvP4XYYplEPXvZqN2dwEIqYcrNNg6VeFq2eFZnwHLZyItLzYWSQnSiDp7fWg9eNiTPJ4OU3dlCfh3ktd9jcZLsJ8E4BuED5ey9/reUKguDWIAheFATBi7Zt25bLyg1LJQWdEwCSIIyxbGHJYPgmkKUKikTTjIBHGoiqPXh1z0fV9UISzVYYgSgFn+SiGdomN/mC9II4g1dVGLwgkBd2DOApDB4toybknVGTFemiqWsIOWrSzcLiDF70fqTerDpn8OT3VYlmp2BH9OCl7Oe8qxX2xI0APLMfJJrCrlpvKWxWLTofS7aR2wMn4NIxXUtnRmW+UH9Oa/t+IK6/bkqrWnFypSI5FK1fv4Lkfijav4w5Gq79RAxenux7t4r2vdWmlHxQKirR9ALZYz80Ek3luQM07rWOlq+Md7rRg9fvDF64LSSXRW5odRPgPQjgck3TLtY0zQbwYwC+0MXfe06WNFmRr0Vz8IREM4FRas9kRRd9aSImQenBI/akUvfgeIEAQK1KNKMDfMHSKBursklpy/Z8Ka+M9uARaP3EvUfxns8/oYAYOcNnRkxWiMGj15MYvK3jhTCDp/TgJQ2sWVYffbZDF021B68uAUJe+Xp1z2vyyXwq7GjV5LOR/kyrD+RxxHjaRmM5ZKOi87FsG/k9xJSBZ9IigyAQ+97xArz9I/fg1rsO5fLbeZWnArxuSjSDQNxbWzUqKAyIRf5GFt3Th5E58rhypFX36I0oNbPzuQDw6HkcBPI+Miy9h45i7gW0FnTuKfsjz9PAVUzrcltmV3rw5P+H4TroGsALgsAF8G4AXwXwFIDPBEHwpKZpv6tp2lsBQNO0F2uadhLAjwD4mKZpT3ZrfYa1khi8qETTNlnP3KHZVXz8W4cBSBampZgEQwadpzF4uiYBzzLvRysoEk11Nvv/3HcUpxYrqb8X68EL4tv6szddjI/8xAtxybYx1FIeosRY0Dqw9Q3LLO98Zha3H5iRDJ4y4yS17Ahtq53I4LHlbpsohAxX5AM0WRrn+dKMphWn0aRSLbmJqSzbBvygs4cYAZZOJaRZq5Wgc08B0IBkpus9YhuTSs1LNIw2YxL4uVa0jNwGIF4QMIlmysyuenxdL8DBc6s4PLuWy2/nVerMezcHZiqQbLkHT0g0+3twv5GlKhuGDeAFAQaPwRtCoK1WkkSzG4zVRpYbab9p5fxjyoj8Gc2uM3g55+ABw8HodrUHLwiC24IgeF4QBJcGQfD7/LX3BEHwBf7/B4Mg2BsEwVgQBFuCILimm+szjCVMVhIZPPliyTbwuUdP431ffgrza3WRtdZK0DkBNXKCBCR4TMrBW6myrJki5eeZmmDcVqoO/sfnn8Q/f+dk6u9FKXiSPqk9eCXbwJuu28UCz1PkeF4gYxLo5kUySxqErVQdVB1PMVlRXTTDgJmMS5IYvLW6i6KlY6JoYl3peawLSayemD/m+oEAePkyeCTRZDLdTjL2em+yIkFb0xy8FAZvI2fOHaUHz2izB4/Om1KODF4QpOcxAtHrzkfN83vWd5m1/BDw6t7veIG8R2YFeHSJDUrI9UYWXbcml8oPU1E/U6u95xtRdM5aRjKrPywlzKP4eIL18nb/PtLLkhJNlcHL9l11f3SlBy/H66BbLpq9mDjsVQ2Eycqo0ktPYvAigASQTBvAesMIDJmtSDQNklpqoYG0ZeiyBw9yYLNcYQwWARdL18VFSUzXYsVJ/b1oRg1dxOq2inUz02UwLPcrmoMXlmiuVl1UHV/GJLjUg+fHXTRdP7QsWkbN8bBSdTFeMDFmm1h34oYrZdtIdNH0/EBkCh45v4bv/cA3cboBu9mokl00O+8H2qiYBMtoLm/0FDDIvrPxEk3aXzY31mlnRpDAdDnHHjw1By9pndRJANcL4Hh+z0B91vLUgUiXc/BooJRZohlh8EY9eOlFh84y9KGYMVeLJJqDEJMggfZzowfPcf3QuQcMhyQPUAz0dOrBa80gqhv7oysumqrSJKfl9qq3u1c1AngDXoR1VMyjJwI8U/y/4njigmgvB09X8sZ0mIammKxIF81lzuCpEk0WdxAIc5SF9Xrq7wk3Ov49uoaTAF6jWVLfj+fgiZgEweC5jMGLBJ27nsLgaWEGj2b2NZ79RyYrYwUT5UKYwaPtLVkmdF2LzVa7fiD206MnFnF0bh1Hzrcni0vKwaPjnwvA6xWDpw7+mjF4JIGNHOcNlWgKBk/nxjoB/vruI/jXJ85kXoZg8CwzVytoGngmzVJGJb5B0H9xCb4Aqd2XaNJETnYGj31uFHTevOQ9vj0Jcz8XiyPBgLhoPkckmop5lApqgWGSaIYZvFYkml4gJ7TyBHiD4qKp9iAOw/kwAngDXiTRTOrBU1+bKEqAt173QpLBrJUUk2AZTJJJkkdd01DgTN9yhSSanPkjZ0PPF4zW0noDBo8e/iYBvLhEk8oytPQcPMX6V7hoJkg0XT8QkkbH9xEEQXIOnhu+gQKMxas6HgN4tokx2wiHntckg6dr8ZuH7weweL7g7EqN/057Zibqza6quGgCnTFa9Uj/QrfLVwB+s4dNlN3thwwyR5HK0EP2f993FP/8cHYz4W4weD6XLDOpcPx9lcGj87bvGDw+adOKgUA75QcBbDEAzP4dYNSDl6VUieYwzJirReeoafR/Dp4wu9H1oQPaajnKBK7o/xwyBo/AFI1PNC0+oZxWIQarC/1yfe+iqTKYQ3DbHgG8Aa8kBi9Jovnet16D33nzVQAYmyTYBT3OhqVVKOjckJJHJtGkHDzVZIUDPIpJUGa0CeA1lGgqLKPnSxfNdAYvLQcPikQzzMLRxbzG14fWOQgke0Iy1rSYBNrGmutjteZivGiibEcYPEcFeMk9eDofDMxzVrPdQXWYwfOhKcekkxuh22MGT30AZ2fwJMsMbCzzRNcYBZ37QYC666Pawv4TMQlWvhJNQ9NS2S91MEqTFP3WQ0QgtV3payu/Y5mtuexFc/D6nb3ZyPKVweiwDLCpPJ+ZGRlKa0K/lhc6Dhu8Ml0sOg5111dYS+q52rDVyrVEBBaZw+nZ5YZ+0B2AIxm8/u7BC03oDcH9aATwBry0BLaO/q/GCbzwwk140b7NACi+gA38jVYAnpKDJxgxU2fmKQkxCdSDJyWaMptsLYNEU50J9wLFSCMB4BVMPQQ8vnNsXjh0eokumuEePKplBXBWecRAzGRFOHDK9ShahjBZGS+YGCswBo+MYSok0bSNRGkcuXlZitlArU0gFerBq3usT1LvHPA4EQOabpfowTPjuYGxzxL4F8eZ2OKNu0k7XpjBc/2AZUM62ZlZ1WQlv4cYY6P1hF5QIHz+0ERM35msEAvZQsZTO9VOT4q4b41y8JoWHTrb0IdmgE3l8/xVNkHZ3xsXksMPwcA2reg+5vp+aJuB4RjQA7KthBRHWVUO0nRmAF00c3w2Wmb+DOZG1QjgDXgRxNATGLyolJGMVtZ5Pp2l6wIgZqmCyMHTBFNicVAiwIjGBo+mrskePCHRlCCjkkGiSRctk2jKG3LSKqsM3lrNxU98/H589A6W3UWMBX0OYIYotGy1KNoBkP1rMYmm48PQtdC+kz14HuvBs034gQRp0mTFTHXRJMc1qlwAnuvBNiTj2lkPXtDRerVaoYysZhLNvnTRlGY8BKbqri/OvUzLUILO85KbUv8asYpp6w2wawnoX4kmk7528XcCef1nHfDQ+hSELHrwBwrdqlZY+kEr9Rzt93NAZbOeaxJNGpcMy/nnKMZwAAN4QQAx2ZxWtPn03bwYdTVXtd8ZPDU2YxgUBSOAN+BF17EWYvDYv1F2jvqwiMFrxWAFSOvB02EbeojBo8+uVMlFk32PZrQdPxADx8WKk3rjCT/82Y1C15AISlWTlTuenkXVYYYnQtapxDoAErxFGbwlhcGjz0RNVmquF9t3xOCt1lyMF4wQmFb/JYlm9H7kCwZPLjcPgFepezANTTzEOurBczeGwbMzDP7oZt9PAC9Roun5ov8zS9U9n8meTb3jHEMqP2jcv6Y+OOm87TeTFY+zkFoLDnHtVCiOoVWJphg49te+66eiY2dmcModtCKm3NL1vmdxVYn7MAxs04rubY7nK32H7JnRDAANSrleEJooNgRgafy9WE9ijqBJrFuuDF7Y7bnT8iPbPwyAfwTwBrwI7GiR18gGXS0JOly4nt9SRAKg9uBpIbdC5qIpe/AABqQILImYBFNKNGng6PkBVmuSNVNLlTr5fhCKO4iWpYDM2x5nLoU114/JOglsUs9dNAdQlWgSy0jbSh+tOr6QPFIxkxUfK1UH4wVT7GsCspW6JwbqSb1Pru9z5lMuN58ePJJodi5Z3KgcPCuDyYrIzIvkHfaTRJMYvEorEk3XF1mTQD6yGS8A7w1KlmiqD0tyf+03Bs/3mUNhGguZ2+8ogeqZw4KjMQlDMFDoVsmgc70nA+yq4+EDXz0g7u3dLE85RzfS7ClLCQbPHD4mVS0151Z1DgWGR6LpeH7IAI7GZM2Oa9wcKp/9EWbacnTR9PJdrhc5H4ZhomME8Aa8CO9EgY8aRk5FVvnrjgfHD8/yZCnqpTM5IwEwgKRKNGk9CqYugs4LgvmTLNK68oBdTJFp0gVcMHWhmdf1ZIBnm5qQft5+YAYAY9pkbxatFwe5HHhF9wHJSgFFomlQnozC4EWYv6JlYG6thqrjY9tEAWMFvq8VBq9sm6kDa5IGqJLR9l00FYmmQxLN/HLwegWaVJOdZjdb4RxG0uE+YPDIkpuCzlljP1ruwbN4jh6Qz0M34PbteoqLpjoYXetXBo/kbyl9hHn+jhwAZvtOdODY6ezyet3FB//t6bbvB/1cElh09zhSPXx8AX/xzUN48Oh813+LcvBMY4AkmgkRPsNUZBTmesHQxiQ4XhCaKNYFg5cN4OU5mQhEGLwcxw55M4PqeAMYDtOdEcAb8BImK5EjSQYEahUtHZrGJZpu6xLNnZNF/NfvvxKvv2aHwuBFgs4VBo+uOcHg8QFP3ZU5eEA6wFMHSn5AxgrJ62YbOuqej0dPLApgVlPCTIXxDJdB0sCVQCsVGcMAciAuGTxy4PRj7qNFS8exuXUAwI7JomTw+HZWHBcl/lqai6ahMKNAPgxe1WHHWURUdHDXImDXMxdN6gngEt1GJaW47O92BtfH5tbwrr9+QLCunRYNJkiiWeHSzFYAXs31UVAZvJweZAwcJctwVABPEyH95qLpcZMVrRcxCW26aBZyYvDuPzKPD91+EA8fW+xoOf1Yvc5fo+dUL/qIyWTFbCGHbKOK7q/DKJVVSzzDPF8M4IfJFh9gzx0rxOBlA3jRnsS8ToNu9Mp1Y7nDyOiOAN6AF13G0b40U3FRgvKZkmVgvc6Czs0WAZ6mafj3r7oU2yeKMARTQjl4JNEM55ABSkwCSTS9MIO3sF7H02dXYr9HDx2Rg6eYpUTLMnQ4PKYAAMYLJmqOItFUzvSiaQhXS5tn9lGFGLx6uAdP5uB5MeavYBpi0MAAHmfwaiqDZ4jlNXLRpGq/B09+r+J4DITr+TF4vXJU9JUZtWaD6yiDp2YuZq2Hjy/grmdm2w6Yj5aUaDIGjoBdKzEJVcfjzqv5yf2kfXt2F81eGetkLV8YWHR3YNZO073M66TBfWcrWBUqgHwmHvqpBLDQ9aY9QnkUTU71YpJKmqzofTdBEi3ZCzncEk1VhSIlecNjqgEgNrajMVNTiSaNt3LuQfO6BPC6xeDlLVHdyBoBvAGvNImmrscZPID14VHQeasSTbXoBmIZeihkXDJ4EjgVEyWacrDy6YdO4A1/ehcOza6GfoMGRgTwaNY+qSyTPURJxjRVssISTdXx0jKwWkt20VSBJ8Uh0LZKk5U4+6kygTsVBo/YxPW6J0xukvqGaDCgLrdtBs8LQrp7K+Si2XlMQu8YPGV2P6O8hBg8eki1AkbJ/GQ9p/4cGtSZnC2jCYO662dmg6qOh6Jp5MrgBQF4Dl7ybL16jvSvyYqMSehuDp7isteii6ahMea808E93YfWetA31uvyxaCqNzl4ksHr/r4UJiuDEJOgGI4MCc5JLEdINOU9uNXru9/L8cIeATT0aWqyEgG8ee0PVfKfp1Q5dwYvyugOwfkwAngDXgRcorAnqQcPYHbrFW6yEjUKaaVo2cQ6RV00Q5EKRrgvikk0PYzzPrU7n54FABznEkeqqIV20KAHz+ISTRqkT5Ys1JSBtLovipYuAGYU4Km1Lhg8uS0Al2gmMHhUOyaLwsyFWJuKwuCxgXX4tyTAy6cHT10f25DA0engRqjKW3pR6vFvBoiiQeftSDTpWK3lxJTQb1uGDkPXQ5MaWRmxSt0T2YlAZy6oVF4QQNeTmWQgfI7IoPP+GqDS4FlP2Ya8yvMDMVnQqkRT1zmz34IkN6nExENO0uF+KtEHpfdIoun0jpEmiaahdw7yu12tTKYNctFxUE1WWr2++71cL8LgkQIho8mKmfP+6B6Dly9wjAHcITgfRgBvwIsu45jJihaXaAJA2TJlDp7ZmkRTLWIUWNC5lNeoLpqAZO8AeeG4PsuL2zVVBAAhqzy3XA39Bj0TWdA5uWgmr4/Nl73KJZZTJTPsohkCeIYAb40AXrQHTzVZSerBA4DJoomSbQhZqmTwXCHbTHbRZI3RZh4Mnh+EGEXT0CV72sFAXXUg60UFygO42aCDbvBqRIimtbauciCdF4Pni7w5Q0coHiFrH17F8VC0DOUh3fl6UUh4GoMXkvgqrGM/Fblopm1Dbr8TctHM9h06b3VNE/mYnRQdg6Fk8EJ91j1k8DoE3VmKnldFq3OQ3+2ix9GwSzRdVaIpJgWzxQgMSrm+HzZZydiDF40JyGt/qPs5TzfZvN05oz14wwD4RwBvwIsu3ijw0bksLFol20DF8bBWc1G2zLZ/V+SN6Tomi5Z4XYsweAR0gLCzYcVxMVWyMGbL988t10K/QRdtgTM4lN+VVATUKHtvqmSh5njSfCMSSi7+nyBTpZ+IxyRwgOf4MWBIjNmOSQZaSY5ZVSWaSg9ekotmnkHnhQiwlj1pHTB4PexfAeSAmkmcmj+couHzajZilsqbwXN8yfQauhZiPrNGJVQdHyUrXwaPBZ1z9ith96j7jAxnXN4D2y+lhkh3c7V8P2hdoqlMKlE+ZidFEs1hZPDUPtuhM1nhLppFS2+p73YjSj0OwPBkwkWrHmLw2GvWEPVcAez+HTJZyeiK2S0GSzWd6kYPXsHMJ2cyloM3BNfACOANevHrOGqyksrg8R68+bU6No/Zbf+sZPA0bB2Xy4marISBhpRortUY4Jkuy++eW4kweIom2uUALynkXF32smDwrIYMHlUSgzfGmTYZk0AAj71f95JdNAEF4NkyVB4Im6wkBUxHAZ5laB0weH5Iomlxp1N6r91yeszgeYpkJIvJSrTn1NK1ltaVBn15DaRdT8r7ogx71kF/1fFQtFp30Xz0xCIW1uqJ73lBYxfNcA6eXM9eSXOzlGQhuzvT6nGZna5lH/Sqfb/NGLwnTi3h4Mxq6vuAZH6HkcELybB7MJ7qqckKP3eKZucgv9sVcxAcErATLTUmQY2GAIaj5wrgDJ4adE4MXpNTPhZ0nlsPHgdilpFvD56XL3CMAty7nz2PH/rLe/uuPaGVGgG8AS/pohl+/XVX78DLLtkS+zwBvIX1OjZ1APCoL83UdWwZUwEe+9dOZPAkC1GpexizTUyXLbFeM1GJpmKy4vsBfB8NXTQBxuBpGjBe4ACPevA0FeDJ0z4J4JULbJ0pMqHEmU5Dj4NVucwwg1c0icGTxh0hF80mJivbJ4q5MXi2IQFC2kzX+760Hz/3Nw82XK7sX+gNm+NziVMWI40kdpcZ77TmWAnkN5BWA2ej66bKNRtVxfEiDF7z/e77AX7s1vvwV3cfSXmfx6ikBZ2n9Db004NOZfC6m4PHe/1ayNvzIwxeo+v4v//L43j/vx5ouLyaIvPuVd178Dxue/xM139HRqFk6xPqtKivuRcMHp2jxOL2MysmLfLzNdjot1KdoKOxAMMCamMMnjBZabx9QeRa7AqD14UcvLyAYxTgPnx8Ad85toDlSnKM1yBU+xq9UfVFSYlmeAD5e2+7NvHzJdvEWs3FwroTAmatlmDwDA1bxgvidVoNAhiFFInmWt1F2TYwXbZQsgzceNEmnOUAb7nq4L2ff1Kwe9T47TXJwQMYwCuaBgqWjprriVkrlc0smvF1YqDQxErV5SHlNSyuMwZkjAM+FSQSMKWi7d0xWRC/Z5u6YAErag9ewqCU6eaZIY2mAdsmCjn24GmhfZ9UT59bETl+aaV+1/F9FHSjwac7L2Ka2P5q/FnXiwM8s0V7chr8VXJ00TQjBj1U1YwGOmSyQsvJ8tBdqjioOj7Or9YS35d5ksnOhWmTAP1kFMGMYtL7CPMqNnFAofBZGTz2r8EZvEbszUrVRcluPIAQEw859YZmqY/ffQQn5tfxput2dfV31D5bgB/XmGVYftVbF012jpZsA37AQIWqrOinkmxWvhlo/VZ0D3N9v2uM1UaX6/l8DMNKSDQzGpXZeTN4CtO22AUXzbwYvKiLJt1v+11e3ahGAG/AS8YkZPt82TJwbrkKzw86kmi+6opt+MXvvRQXbCrj9KJk3khCScxYSKLJ/+9wF81ywcCrd2/H5dsnUKl7OMCz8B45voh/fuQUrt87xb+nCYlmqosmN4xZrjooWDoKPDaBgr1DOXgK6KT1K1sGSpaBlaor3D0X+cyNBGZyGdfungr9Pi1zJzeOAVgfHs3crjtRiWZ4/Wm21zZYT2PJMnJz0QzFJKTcCFeqblOGQAV4dbf7gxXqFTP0DA3iCQyebbQm0ZRSuPyCzsn8J8o8tybRNMT5m6XXYG6NAbullJlHdV81YvBs7kxL1U9GK5SJqWtaV5kRkoIaWna3TjWyo2gZDfstq47XlM2tbACDV6l7ucWFNKoYc+QHsLp4W6k5vevB8/g5Ss+YqtPPAI/9O+wSTdFm4AYCxA5lDp6uMnjUV9n4e6qjLZDf/lDz5brholkw9dzcpQF5PtD9Nq8J342oEcAb8KJxY1pvWrRKtpQMdQLwdkwW8dtvvBIAknvwhERTAXg6WfX7wlXy519xCQDgj7/2DM6v1uB6vmDOlisO7xXSuEQzPQePAssFg2eGe+DCOXhxiWbJNkXfHAGxGIOn3DSv3RsGePQQ3z4hAV7R0lGpe6i5PoJA9uUZWvwBSoHKr3zeVkyXLRw5v7SOkwwAACAASURBVIb1tfYGdK4XYLwQlmgKiV8K4FmpOk0ZApXB6QWbI2R4CT2L0XJ9P1Gi2YprqDDEydFFUzVZUauWUaJZdXwO8LI/dM+v8uunmgzwPLqOtGQpFk0CFK0wwOsriWbQK4lmoMhZs31HNXYqmDoW1pN7IQEG3pqB/Y3owau6Xk8ApWrPD3SfORISzYzXX7sVBIGI8qDJv5rjASWryTc3pp4rEk26tzmeHzv3+uj21lHV3UgPHv9v0zYHYjTN1lyDm5UrgJjRFRfNgmnk04MXYXQpK7nf+2cb1agHb8ArzUUzrcqKa2UnAE8tVaIpevA44Epy0aw5LK9OXZcdkwUEARucLq6zgelK1WVhwVwe5QfxgbJctmTwipzBA+TstwrwkkxWyrYhpJtkskIMCMkdVBbmuj1hgEef2T0dYfBcORNe5r+bLNFk7OTbb9iL//kD18A22rdX94NwD55paEJ6kwbMVqouKorraFJFGbxuF5kUaBn6nzw/fm6YLeZP5e+iKfOIosxzFhdN1/NR95iLpuihzPAgm+MAL53BI5fdZFaKWMKSHWYbesF6ZC3fZ5NbTDrZzd/hMmEt+4y2auzEpOLp+y0bwOu9i2avGLxeO9cJk5Uuj+bpsjI0CfCy9t1uRMVcNPt3VTsqcoJ2fD+RPR6Gcv1oD17GmARiNGkysQs9eH7QheVaei49eFGjIZrgGgG8UW14ZWXwugHwppVZyRiDp0oFzbDTZQjgcebr3HI1DPB4D1YQsAs6bTNp2StVF0XLECxdRYSVJ8ckkN68bBuCbSSwRgNkFZhRbVVALQB87xXb8KfvfEEI+BUtgw+UXP4bEihGZWXE4Il1tIyOevBU8xhVopn2EKMswkbAo9dsDtmMpwVyq+X5ftxF02jNZEW4aOY0sGUSTTIjal2iSdp/1WQlS95PJommxiZjkgbUtM+KEa1cPzF4FCLdExdNClRvw0WzkYNiEASoOn5TsE/nQU8ZPMcLmVR1q6Q0sDeD7F714AlzL12qWLL23W5ExSSaQ8rg1UMSzXDPWT+b4LRSrpeSg5exBy9vyaqrADEgv3Mr7x48VUoKyHFAP0/MNKsRwBvw0lpk8Eq2VOXmBfBU4EP/kyYrCpPEP0cArqysC7lPnl2uCklT3fOFRI/+TpdoEsBzULCkRJPApK1IFpJ68NRwcgJ4i+sOCqYuZXYNQHTBNPC2G/aEgDZlDhLIDOXgBQGWq4646bp+EHLp7ITB8/wAdkpMguP7ePzkUki66Hq+uJk1Yq8cT8oge9LDosrwmmX4JDB4rQI8aWaRE4PnKQxerAev+XrReVO0jaYuqGoJiWYleTtIdqjrWqJ1Nj3oimb/ArxW5LudlHAcbdNFsxGDR6836/GoiqiVHjJ4Per7kyHIvQkX7pVEU/ZhaiITtZ97eWT/1XCxWdFyxfNWYfAy5sQNSqnPHUA+F5sdUsFgmfkCMcngGaG/81xuHjl4UYBLlTWzth9rBPAGvGhMmwZ8otUNBk+tqMlKkmPlciWBwePukzPL1RDzoGuAIUK64yxNdNlVx0fRlBLN+TXqo5NgUl2nkESTP4jHec+d6weh79HD4fLt4413gvI7VUeRaPLt1TQNq1UXN/3B7fjid08DSGLw2gd4ru/D4sCItpGWfWaxirf+xd34x++cFJ9fVQBNo/4z1wtEMH0vBvuez46/riUDkfBnE3rwjFYlmvkyeI4ykxqPScjA4PHPFE09sxMaAMxx90x1AkGtQJVoNurBi0g0+8pkhVw0u92Dxx1HW2Pw2L/MYCOdwaPXm7m0EfMT7ZFdXK+Hrt08i8BIt0EJyzZVB6G9YvB6A/DCEs3+HShSJA3dZ4aFzVLL9wNxr6grGbnDFnTOnv8qg8f+zeqimbdEU2Xa1L87Xm7OOXgiJkIPw6J+vm6b1QjgDXhpnDPLiO8EyBhTAE0u6xFx8ywkmKxQL8tSJc7gbRkvwNC1EIMHsIwaAnWO56dupypJLFpGDOCNqwAvyWTFMsVMq7peKgi9eOsYfuamffjEz744dT+oRQzeeozBY3Kr1ZqLY3PrCAL24FGZUMbgtemi6UnmC2AzlPT/mZUqggB48Mi8+PxKVQ4SmzF4BHh7MdgPAumi2ZTBC+IyyJYZPLc5i9lKqUHn7cQk0IOlpDB4WWZVqQcvCICVBADg8cFcmkGJkGhGMiL7KeicHAq7zuAFkilslcHT9cYTNTQzXG8ihaymsGk//7cP4fe+uD/TOrVavTJ2oeOot3B+d1LSRbM3Ek1d0xSJZv9cP9Gi85wmioeFzVLLieR7Clv8IYuGcCMMXtYePMHi5p6DJ90uAeSWhef5bDxo5e2iGXnujRi8UW1YCRfNjNlBBGI6CTlPqs08s05XWCMgnIMHsEH3UgKDZ+gatk8UcHapJiScgAxkBpj0oJnJCsAAHP2uZPAM5X2FwUvowStahgCqKjA0dA3vfes12Lup3GBPyGIxCT4qTrwHj2q15grpRJTB66QHzzQ0JatQh6axEHXatw8fXxCfVwFeI/aq7kpjnN4weNkH157vx4xMWu7BIwYvVxfNDiSaBPBaDDqncx5AYkirGMylsFIEAO0owOujASqdG7revZn3IGBW6p24aBZN1kubNBuusmONZolVZlldzunFCk4vVbKtVAvl+YEA812XaHImlu6JOZrsJRZtV7fPZTV/laRp/cwEeEocCDA8bJZapOaw+XPB7xKg2ehyPF8omoDsAI9YW5kLmM/6SKaNlFH5XHsUB2HmpOKQOYBRx+v+vW6b1QjgDXjdcOE03nTdTly8dSzT5wlkdBJynlQEGGMMXmSQaCsATwVdALBrqogzSxURTwAwtksCvOY9eECYwZvLyOCNFSSjaZuyZ61st89yFnhMAkmrRA6eAkRWqo644YWMYHgPXjtSGRUYAXIbTV3u+6Nz6wIIrCh2+o36zxwvEPuxVy6aujK730gyEpW4Agz0tyIHqeXM4DEXzbjJSsHUMz00RO+mpQSdZ+nBW6uJY55ktKJmuyUzeGy9o/uzn4LOfQ68kjIl8ypplMGAZNZrMeqiCSSznyrIbzT4V2eQ15X/r9bcrjhdquvSbSdNBqCzW7l3WqIHr8v3L3EOaBggiaamSDQ3eIW6UNR3XrINOF4Qj0kYko2O5uA1yjxVi25ReQe/q26XWdajleWSUqkbLppUIwZvVBtWu6ZK+MhP3BizNE8r+lze/XfE4NGDM43BMw1NBIiXrHAM467pEs4sVcX7AAMlMsOtQdC5CvBMRaK5mtCDR66YmtRblywzEeCp32u1KOi8EunBUxm8laoba/YG5H5rZ1DNbvC66F2k5ZoKgwcAj55gLJ7ax9Oo58bxfDFB0Au5nh8Bqo0ewCLbTSnTSGdBl9adWE6cypTk0YOiBp2rzDZJd5sVSboKLTJ4c6t1XLyFTfgkMXjhbLeEHjyP9XASOKUJkf4zWWGDl25JNFWgpmvNjX6oogwekGzqoZ4DzcLQaQKLohKCIMBqzc3NEChtvboN8EiiqWnNJ3HyqF4FnauTA/TM7baxSyflB7KFAhgeNkstemaVI8dDuEYOyTa7XhDKwaPHYrPNi5qM5HUOdK0Hj49z8mbwzAjAG7lojmpgqiwAXqHJJ1urTWMsKmFhjVwr4zl4QFiiGWPwJos4vVgJm6zoUmLAGLzk37dCPXi6kAMQS1VKcM5U+9RUk5WCoYubXCcMXsmiHrywRFMFqas1NzQYoKIBXTu9IjSzpUo06d/FimRHHzm+CCDag5f8e74fhExnesPgQUgJAeB/ffFJ/J/7jiZ/1o/Ld+0GEs13f+ph/NfPPh56rep4LDpAkah1Uq4XSJMVfg4XDL2hdb5aKoOXdRa27vpYqji4ZBsDeMkMngQtSYtz+b6ka6CXrG3W8pqwkHmUkNm1+Dvq9VxoYJGvngONzoea44sJuTXFutsP8mOb1VIneSo9cNEMSTR7ZbLS5Vl51UWTVCz9zASofbnA8LBZajmRfE96tuYtSdzocnw/JDPMamAkJauD46LJxnCd5+DVXV+MFaISzX6+bpvVCOA9x0oCPKvJJ1url1y8BQAwXWbLFQxeRKJpKazKdCnMIu6aLnFZonyNZmgAHpuQQaJZUHLw5tbqGLONEKgqCAaPDWLffN0uvOzSLYKpCDF4dgcMns1dNJ2IRDOFwQtJNC0CeK0Pql3fDxmrWEKiqYnZqJ2TRTx2comvgwQBaT031KBOoDxvuZ7vB3jVB76Jzyrunn5k0HHb42dxx9Ozid8nMKiW2UCieWqhgpML6+Jv1/Ph+gGmOROdRx+e2oNHp6dt6ihaeqZZQdVkJevAiwyKSLIdZSkBtl81LotLZPB81sNB4FSA+j5i8CgjUesmwKNBeqsumvxjRhMGLwzwkvctTTZsGecAjzN2KzWH/91diWY3lq+WMLHpkT1/vUcumuKePigumuRKqw2vi6YbZfD4OZA3oNnI8nzWN6yyUEZGdlxKFPNlNLvH4Pm59eD98EfvxQf/7RkAcYlmP1+3zWoE8J5jVeoSg/fvbt6Hf/mlm3DzZVsBKDEJMQaP3Tyu2DGBqXIYZO6eKor/q66cBM4y9+ApMQkL6/WYzJIGXQYfIP7FT7wQN1+2VbB8am5cJxLNomXAD1ienq6pzKH8zGrVFTc8VaJJ29MOayIZPM7cRZg8ALh0+xjmeSC26rSYNqCjRmk52M/3prdaZ46iT59bEa8JIw2+W5YqTohtVCs5JkGHk7L/lqtuSApMckjJlHTOXLCZVLbPVYlm0crI4FFMgiUnOZoFnZ/nEQmXbGNRHmk9eIboX0uSaHKTngiD10yi6Xg+fvDDd+OrT55t+Lk8irbB0NvrF/r9L+/H3957tOFnVIlmKwxeINgbNGTwskg0iWWg81LkVfLrtBsxCWk9f90o6gcVfbY96sHrNhstXDR1FeD1zwRJtNR7AiD7sYap6P5V5q0hBPCGSaJJ26i6aJL8uXmWrDShAfJjNIWLpujBy+fkCvXgdXjCHp5dw8GZVQAjgDeqAa5t4wW8+9WX4U3X7cx1uZqm4YYLN4m/k2ISAHnTuOmyLbFl7JouhdYTkIMrAHDcAHrKGWuZYYZOlQOMRwGelWxdH+7B4xLNQvsSTVreqcUKNo/Z4kYb7sFzFAZPZSE7YfCCMINHRh9KT9628YIY/K9UXViGBk1LZvCW1iWwEjl4br4Pw2WxLhKQCJMVxdktiZGi95IAXj2FaVypOiHwQ5Kt6EC6k1Ltqk0F4BUsI5NlepJE0/UCxjamPNAoIuGiLWUYupYI8Mi5MI2Vot4GK8rgNVnnA2dW8NjJJTx0dL7h5zqtIAiYyQo/x9uZeb/t8bP45tMzDT+j9tKxfZVt2apFfqFRD15ICpl8vtHrWyITD6v8elSlRXmVui7rXcrZo/J8MsuRf3ezNiIHz9A12IaeKRplo8rz6Xqivwcf7EQrJtF0ohLNwd9muheoOXj07Gi2eTGJZu4MnhH6u+Plevm4aHo+62emybI4wBvc2Y726YlRDWRpmobfesMVXf8dYsNKEQbv+DyTxd186dbYd3YpDN6eTSXMrNRgqCYrfjqDFzJZUSSaQJyFI+AVBQSlJJOVTiSafHknFyrYojCmIRfNmmTw1PsK9TC2OtPsc4mGoUhbVYkmwPbHVMnC0roEVRNFC3XXT2Tw3nnrfbiUh7tTH2Et55EYAcjlihxQ+n5YvqV+LlqeL/vdqMq2kQhYa66Hmuuj7vlC6keASwykcxjYMokmZ/DI0dTQUbL0bEHnLjF40pTlHx86if/82e9CA3DrT70Ir716R+g7c5yV3TpewGTRTGbw/MZ9ZS6XlrbK4D3CTXtmVmpNt63deuzEIm4/wIAZmXO0M/O+Uk1ng6lCLppa9hl+lfkrNurBU67t1DB0wSyz+wdJh0miSa9NlfObq1XXq9smKz6Z5fS4B0+99rtRUdl9IeM1v1Elgs57dBw2opw0iWaP5MG9KFLahHPw2L9ZXTRNnU325u6iSRLN3HLwAhiGBqNFt+xorUaeA1a0B6/L98Bu1ojBG1VX6ro9U/i9t12Ll18eB3IA8NJLNsde2zpeEDfb3ZzNU2MS6m5chkdFNyWATFZUgBcGmQLgRcAigULbyCcmoWSzZZxaqIgeGiDcg7dacwUbE2LwzPZMVoTc05DAiJqGJWg1MFWysFJz4fsBVqouxgtmIiCquz6eObeCg+eYfEH04OU8A04M3nKEwYsDvHQGz4zcmKdLFtbrXgwk08BeDQKvdoHBcxKCzluRaFbrHjQu7aVz9dETiyiYOnZPl/Bn33g21itDDN6WcRuTJSsEmKlY3xMEKxVdhuPLmVEgO4P3KDftme0iwPvco6fwZ994FgAEMGiVwfP9ACs1N9FhVC1PSC1bYwr9jAxetd5coknniejB49enOhHTrpz4a/vP4Z0fuy8GXEMMXg9y8EJB510cZLseC5SnibeHjy/gz/m5lHepJisAMl/zG1VRieZwArznAIPnh/sKAXkOZpVoahzo58bgUQ5ezjEJebloRlVBo5iEUY2qSem6hp/6novEACdaE8W4yYuha9gxyVi8PQTwIkHnWgqDx4K8KXvPCPXkRSWaBJ6is7chiabIx+ugB49v+/nVGraOSwaPtmfreIGBDA44Qj14Zns9eKo8jH6HmC0CQGMFE5MlS/z2atXFRNHEWMGMuWieXFiHHwAzK1XxXSB/ww3aByqrQnb+UUCcZACQFJNAPZ5RFksd2BOLKQbSLTJ4DxyZT53hcz1fHNMQwMvqoul4KJoGNOVY1j0fOyeL+KXvvQyPn1rCfYfnQt85v1qHbeiYIJY2TaKpqc6Fkfe5+6epTAiw3278EH3kBAN43WTwwg67WlsxCat1F0GQbECjFrWK0MA3u4sm/57K4CUcb3XgkGahHz0vSTK5qjB47bLNDxyZw/1H5mNMZi9z8MjcoxcMHt2zJkvsHvaZh07gj7/2TFf68egc0JVJx36WenlBELrPDAObFa0og0dMNT1rh6HvkMCUpYwlshrnBIrywMgxXzTqopmXRDPqotmuMVAzgNfPEzPNagTwRtXT+vZ/fQ0e+G+vSX1/9zQDeCTXJDt3gMmcoq6cahGoK1psUEyfLdvJEs1okPNL9m3Gj9y4F1ftmhCsVycMXlH5bhKDd/HWMgDpfBhy0eTr/p1jC7j1rkOZf5Nm6UxdSuykRFOC1qmSBD8rHOCVbSNmi35sbp2vI4+24PsydwavGu/BkyHIcr8wa/j4DdcL4kHnchvrodfVAS3FRtDga1MLDN7sSg3vvPU+/PMjJxPfd5Q8IsmmZnfRrDiemG1W5aebx2y844V7sHXcxie/fSz0nbnVGraMs37PJIAXBAHfr+n9Nq4flmgWeE9qI4nm4nodR86vwdC1rjJ4S0qOozCKafFUFGxxArsJAB+78xAOza6GXDRbAZLq9wSDl3C9VBNMVr7x1Dl86bunY5/ZwieI6NxfDTF47Q1A5nmkDd1/outiGVpHAC8IAvziJ7+Dbx5I73WUAdvs724CCwLRk3xy8ewyO08XI/eHPErtwQOQeVJno4rk8NLsZoNXqAtF4Ee0GfDjYerdn1zoVUmJZtxFsxmA9ZRzVtfz2x9RF838GDw/pDRpd7HRCa6oRLOfr9tmNQJ4o+pp7ZwqYvtkMfX9XVMlTBZNwfCpeW5BwOz904ouTJo1L6SwcPR+lPHZNGbjAz/yfJRtUwyoo+xfK6X2H4YZPPYvWdlT+HgoB4+v+9/cexS33HYAM8vVTL/pecosHLloComm7KlSAd4y78Ebs81YD96xubXQ3ww8d4/BW44weKrJjvxsnHlxvXg/DUUeREGOekOn96ImK1lkb+eWqwgC4OxS8rFR84hoG1qSaDq+OIcM5aGzZbyAomXghgs34fBs+PjMrdXFZMJkyYrNTtJDsJFzocOb16lR3zZ12A1C4wEmHQWAl168GUsVp2sPRdX5lHrjGg0YTi6s460fvlsw0IAEdhUnLt9drjr4g68cwL88fCrkhGi0ACT9gMVQaFpzBo/ubQSqPnbXYXz49oPiM1UBSkyYuiYkk2rfSLsM3iIHdvNRgFeX10InEs2VmouvPHEWdz2bHG0CKFmXKWxynkUge5Lf++ieqk4a5FXquQPIyJx+LZmNSX8PPtiJVloPnpBoDgGqJYmmFXLRZP82O6bR+11+eXXcRVMwePm7aHay3KhUX1eug35n3pvVCOCNqq/qZ2/eh995y9VCFqZKDQEGENPKUhg8QObdjUd78Mxkk5XQslLYv1ZKjYggiRUggeW+reEw6jCDx757hoOHbx/J5kzoCg1+POicQGuZ9+DRb6/WXEwUTJQSevCOzq2H/i5wA5q8AV5DF009CvCS+8rSGLzFyABOBT30Xi0SkxDNwdt/ehl/c8+R0GsUSXB+Nc4ARPOIaBsKLcYkEEBQt43OpT3TJZxarIS+M7dWF4Y+m8oWFtbC6ybNH9SZ3SD2GdPQBYNHkuVGDN7dz56Hbeh47VXM9IX2Td6lgnWSlDUauDx2YgnfPbmE/aeXxWvq8Y9OFlAP49xaPcTCaFr2nCzPD2SwfRMGj3JD6XyYW61hTjlm9HrRMlC2DTEBo4K6dqMSiLmLniMVMdlR6IjBm+EM2fxaOkMWzbrsJrCgfuaJIrunn+UAb6ELAC+ZwevfgaLHJyXS7gnDUPUmAG8YcvAEg5fgopk1B4/aIvJ20bRzZ/DCveLtLjc6nlDbgjaV7VEP3qhGlVfdcOEm/OiLLkCZM2emER7g72oA8GT2XmMGT+e21Y0Ani361TowWVEBnsLg0XpdspU5U9JMelIPHtV9h8K9VmmlurelxSSMF0zRn7ZclRLNsYIRk3tFGTzT0FBowua0U9LsxBfLli6akc8mMXgJjnjTpeQePPX79B4NpDeVkxm8v3/gGN77xf2h7SZgR3mCakXziFQGr2DpmWISqnVPTBKobDMxdLuni1ipuiHAQhJNANg2XsTCuhNaZ9X8gZYdZYAc3jtI541lNGbwgiDA1546h5su24J9XHacVx/ed44t4DMPnRB/L4Ykms0DyMlVdE4B4eqM7XLk4U7HcmGtHruWWnHRlOYa6WZJVcdHyTZQMHUxiJhfq2N+rS5+i0BB0TIwXbYFKFNBXbssG+3LKMCpOszcZ1PZ6gzgcda0IcALwix9VyWa/PwldYjc/vwlmkkumv08UPT9sNnNMDJ4rjBZYeMBuudnBUCDUEk5eHLypPF3Qz3HupZb2L3HJ3GIVcy/B6+z5UZVLqqiYLps9zXz3qxGAG9UfVkqg6cCn0YMHvXg0aw5AakkmWXB0tEA3ykumvlINNUevB94/m589CdfiEu2pUs0o72G3z6cDeCpoemSwYu4aCoSzcV1zuAVLZRtM5Z7dWx+PbT/LaM5m9NOqYNuAmDkohmV0kYH5QB7OGdl8JIkmmRjP1YwMVE0sbBWx7nlKt75sfsws1zF6UU2WFWlfsRSzSUweLR/Yi6aho6SZaDu+k0HFBXHE+eQum1kmU9Os2cW5TrNrdYFw7dtgn1uTgGg6iwtnX8HZ1dZthxfH5c7kqrZfY1Y20Ozqzg2t47XXrUD2yfY9ZlXH94n7jmC931pPwAGJNV+yiwB5OcFIyfXRz1/ovKcOQHa6yEw3KqLZpTBS2JvKnV2fEu2gWrdg+v5WKw4obxHAgUly8COyQLOcdZppeqKc2s1IdokS82nMHhVft6VbbPrDB6ZI5F5VjcH2TRBMVkM39O7IdEcSBdNVSrb4e39iVNL+Om/fqCvtlkGnYcZPBGDMvj4Tjz/VYkmPTqaumiKcxZt54umrZOpxF15OcUkuJ500exkuVEGT1MUBZvKVl9PzDSrEcAbVV8WAavoAL9xD16UwWM38iQnzKJlxHLT1BIOgp0Endty+VuVHLzpso03XrtLAE/qKzITYhIA4ILNJRw5v5ba66WWGpquggq2fG4cU5ASzVOL6/D8AFMlC2N2mMHz/AAn5tdx5a4J8VozNocqCAI8fHwh8yygepOl/wsDhlgPXhzgucqgmmoywuB99jsn8cY/vQtLFUfEDxB7KpkSXUgf7z8yj/uPzOOBo/M4zaWQ6jE4z0HMXMIAVkpl4i6aAng2semvKiYrui5jQLYKBo8BPFq39bqLiuMJtng7B3g00AbkIMbQNFy+nR3XgzOr+M1/fAw/duu34fmBtJ82ZGyIberCZjxaX9vPTDRec9V2ASrzYvBmVmpYrrqoOh7W615oHahXohHAm0sA4WEGL3wMCIzMrdVCToh6C3l7nq8wNw3iTqouY2hJvrew7ogwYjqnpERTx/bJojiWazUX2ycpG691Bs/zA3FdxHrwOMAbKyTnSKZV1fHw5j//Fu45eB5AdgZPleF3tweP7Uu6L1B1h8Fj/9I9qWQZmcLVnzgVlhP3qjyfJM/87w4H93c+M4s7n5kVJl39UPEePHY+kER4kCSa0UkZKopcUp0gs7po+mLsQBLNPNY0P6Ytdbl8W9tm8CpxBo+u201lO9XheBBqBPBG1ZdFwMrUW+jBM8Oz5pS7kgTwCqbeMOTWUiIF2q1iCoNHNc5nkolhUvGmGi/xozdeAAC47fEzTX8zicEzFakdAIzbJkqWAVPX8N2TSwAYiCwXzNCA7sj5VThegOfvnRav2YYOy9RSB/tU9x2ewzs+ci8eOrbQdJ2BaF8UW4ekHDz2flJ4dxD7nKFrmFDCvr99eA4Hzq7g0OwqxgsmNpXtmESzaBrYM13CyYUKTsyzwcmxuXXR63ZmKc7gJfWbSalMPAePZManI/1z0ao4fugcouO5WenBA4DTS2w5IgMvwuCpbJqadbRjsoCJoomnz67g9gMzeODoPP7m3qNwPR+WoYlroCBMViRIeeLUEv7XF59EEAS485kZXLN7ErumStgyZkPT8mPwyAhjZrkWk9pK7y/NhQAAIABJREFUKVH692mfqH2S6rkWddIksDMf6cFricELAjFrTnLwNAavyBm8iuOFgNB8BOAVLAM7JoqCwVutudg6XoCmtWeyslyRYHIxZrLii56/Vhi8E/PrePL0slAbEBidW6unDi7l4I//3c0evIiLJlWziZZ2ShpWsL+LGYPO//vnnsBvf/a7ua9Pswp4NqZgUjs8DnSens1oDtaLiubg0TWp61pLEzgbXQ8fX8CN7/sanj23EnvPSejBI4DXNOhcddHU8mPTXY965brgoml0pwePxobTZQt1np85iDUCeKPqyyIGT53dnSpZDSWTUZMVYq7GEqIOipYR6+1Si75btjrvwWOz4fH1HrcJ4FEPnlwhtQfvtVfvwEsv3oyP3nmo6SCBHKuSXDTVHDyy0X/8FAN4F20Zw5htwPECHJ5dxf/43BN4y4fuhqFreM1V28XyLVPLxOA9c5Y9fJ7lAenNarnqCuaVBuCejxCDt4n3Da6mMXgJgH26LKMCjnPA9uTpZUwWLUyXrZjJStEysGcTY/BOiM8viYdAiMHjoGGl6sYYGocay8lkhXrwDAO7pkqxZSVV1fFCAM+IALxt4wVYhiaAIgFNcmwVAE8BoOosraZpuHz7OL7+1DksrjuYKln4o68+jbnVOnevlS6aUVD/uUdO4RP3HMXJhQqePruC6/dOAWCAdsuYnQvAC4JAMIEzK1VxrAggC7e3LD14qkRTAXUxBo8f08WKI0A6MYWtuGiqk0cFU0/pwWNMGfXgzSnHif4vz0sd2ycLWKt7WK25WK25mCxZKFvxvtkspbJWUYatys19kiTbjYomQehfOnZ1108Fil4gB9hAdyWasgcvfC+OAtw8KmayklGieXJ+HU+dWe65tJEC542cjgPd285lUJ30qsiAjJ7FdE3SBM6g9B0+dWYZfgA8dTYO8NwEF82s2Ya+mPzLd394vg/D6NztMr7cIEQAtO2iWXVC7TSqiyb15PeT1LiVGgG8UfVlEYOnMjiNDFYANQePSzStRhJNPSbpU8sydBRMPZQn02pZBtOHJ7F3ALuRjBdMMYOcFJMAAHs2lfBrr30eZlZq+NQDxxv+ZhKDJyWa4Z7EqZIEOBduLgvw/JYP3Y1PP3gCP/j8Pfjqr70CL714S2ibsrhoHjnPzFkIVDWrlaojJIfE0DFWTu6XHZNF6Fq6i2YiwCvZYgCnMnITRRb2HmXwCqaOvZtKWKm62H+GSaW+fVg6mCYxeACwsBYGCm7UZCWBwTuz1ITBq3soWfE8IwJwuq5hx2RR9AcKBo+fb/S5sERT9uABwOXbJ3COv/8br3seKo6Hs8tV5qKprLNthPsuj/F9ec/B81hYd3DZdinj3TZRxOxK5wO71ZorgMG55ZrIK3vejgmxPwy9cQ+ecMWMMHjUhxWV5xDYCQIpkxR5e224aALsPpTE4FUdH0VLFxb6qtRXlWjqGruGd0zS8axiteZivMAmjtph8FSAFz13KX+xbBtYd7zMMmu6Nqgn9JzC3qTJNAPOdqaxDPtPL+PuZ89n+v1mFXXRpIpufx4VNVkpppwDatE54PoBnuATb70qz8836JyO/bk+YvBoUpIG88To5i1J7HadWmDPjRMJz9aocgSQMQnNLmMh36f9kWsPXudMW9JyDbUHrwMGb8+mkvhbddEkl+NB7cMbAbxR9WUVTUM0u9JgqZE8E1BdNJubrBRNo6GL5ksv2Yw3X7errXUP/Y5lhBw0ozVRNBNdNCn/b6pkYbJo4WWXbsGL923CJ+45GppdDYIgBBRcNQdPMHdhJq/MwTP1omybKGCsYIrehKrj4fPvvhnv/+Hrcdn2CZRtQ4BE22AD0iSZpFpHeO/F8fm1hp+jWq64QnK4HJNoss9MFi2MF8xUF82k40lh3zXXwxllsDFZtDAdAng+bIPJdvdMMydIGmSpg1N1wHJ+tSb64aIyzbhEk71umzq2jBdg6loILCaVarLCliH7Aqh2K1EJxFLR+WabOjaVLcyuyt+RjfQc4O0Y58u08OorFKZWl0HnBOrVHiJyV/0iD+V+Hl8OwM6nPHrwzinA9NxyVYAx+i1iIRsBL2mEozJ4DnZNlaBrcQZPBVnEQtIkU9YBRFYGj45vibM76joSk0gyTk3TsIMb2JxbrmGt5mK8YGKsYLYVk0CgZvdUMR50rpi/BEGyQUxSnREyZvbv7EpN3FOS+lQBCYbTYhI+8NUD+I3PPJpxqxpXNAcPYPfEbgSdR6+zImdpG4FlVbJNuZK9qiCg64n93ekY/GwfSzSjMQmaBm6yMhgMHp0nyQAv3PsNKAxexhw8wWjm2CunKrHy7MELM3jtAjwnRB6QokDXpJx7xOCNalQ5lq5rKFsMhJFysRmDZ/HoA5JqpsUkAMC1e6Zw5a7J1GW95frd+ON3vqDNtZdVtAxsHUtm8AAGPpNcNAG2/gR6AOBdL9uH4/ProeDgD91+EC9//zdxlDNmdJMO5+DFJZqAdJm8aDMDNBRN8aMvugBXKftG0zRsGrPEMq7ZPYnHTy4JliqpjrbB4O0RDF6yycpE0cRE0Upk8Lw0gFe2sFhxcHKhEprBnCiaIYlm1fFEzybN5vlBmEm9ZNuYGLx6foD5tbpgk6jP6A+/cgCPnVgUD1oRdK7Lc9LgzFsjgHdmqYKlihOaWTQNHZNFM8zuTpcUiWa4Bw/gYEsBSrQPaNLksu0MLN140Sbs2VQS54qRaLIi4yvIPIHiO2g/AMC+LWUcnl3reICgOpbOrNTEsaLf0jn4T/uZuutjuepC14DzSh/YcpXJUSdLVrwHb60uFAAE8HSd9WS0k4MHMLVAUqM+MWXE7syv1aFr7J4gGDxXynS3c4OpmZUqVqsuxgomN0JpX6J56fbxGMCj3xzjjH5Wo5XT/Hw+vVQV8tordrJjlWYK4fF4Exl0Ht7HB2dXudFO5yybAHhKD94Fm8sxl908is592q5iBFQk1WnFDfeRDgBeEAS49a5DsYzMRtUMaLdSrueLa6efGDxXmKxEYhIyKAH6qei4Jj1baYJXNVkxUq6taEnnVwZ4czK7FECMxh+5MnhG5719y1UXUyVLEAE0oVeyDHHdjgDeqEaVc5ULJgxNXsA7J0sNP28ZGorK4Fe6aMb76N771mtwy9uvy3Ftk+vKnRO4ds9U6vsTRVPMPEUBis3lglRvuGYnto7b+OS3jwFgvWF//o1n4fkBvnGAORnKZUkJajTofDwK8LYwu/wbLpjGq6/Yht943fNi60mskWXoePG+zVirezjAewAeP7mE+5UYh7rr4+SClEM2k3jVXA811xcSTWJqoiYr40UWYZAWk5AkuZ0qWVhad8TDkAbvkyVLsHu0DjSQVkH1i/dtAsBmRJ+/d1r0ljATDogB7NxqDQfOruCjdx7CR+44GAucFTl4/Fjsmio2lGiSLO0Vl28Trxm6FmODd08XcXapCs8PMLdax3jBDPXtbZ8ohnrwhPkD31VX7JyApgEvuXgzDF3DBRzsW4YGKyLRJInTzEpNDFT9gJ3D5NgJANfunsJqzcXRuWzsbVoRMNU0JkskKfONF22CberYPVVsGJNAzOu+LWOou77oVVupupgsmZgsWokumpduY6BXADyNHb+sgNUPopEnRmoPXsFkDF6Fy/M2lW1sHbcFwKvUfXFPI4nm2aUqVusuJgomynZ7DB6Bmku2jnH3TrltqvkLgMwAkiYa6q6PE/MVrNZcXLmTTRSlMXh+EAYW6pxR1fFwksvRDs6s8veDxH2ZpWoJMQkXbxnrCsBT2RCAKUYANHTko/13/d4pPHq8fYB3aHYNt9x2AH9//7HM38kz6Pz8al1MuvQXgyf7WYFwTMIg9eAJieZCAsDz4zl4WSNIPGVSolm+aCvlcSDWdQav7ZgEB5MlS9wXSJLPHI7ZuZJVxdBvNQJ4o+rb+pmb9uGN1+4UrEIWBk8d3BIjkyTR7FV98udfil9PAExUU4pcKJrj9pbrd+P7r9sp/rZNHe988QW4/cAMvnNsAb/5mcewaczGhZvLuONpBvCiMgtABXhcoskHbhLgsUH9BZvL+MTPvkQwBWqRFt0ydLxo32YAwENH5xEEAf7jpx/Buz/1iHiAHJ9fhx8A1+yexErVbTqAIkZuusyiGgSDRxbqCoM3WbTSJZpGHOCRDPM4Z5xewvsJGYNno+J4DGDyfiiAxRAQ+3vzZVsBMHnw7ukizq3UGJjicsgriMFbrePr+88BYBbhBBzoQasrEk0A2DVdEgzectXB5x89FRpkf+vZ89g6XsCVOyUzZupaiJ0DgKt2TcL1A7zvy/vx1JnlWL/ntolCyPAkms+1a6qEz/z7l+FdL9sHgA14ab0Fg8dz8GiARMDt+Rcwd9Xn7ZgQgwgAYkLj8YQ+opmVKlaqDuZWa3jNB+/AQ0fnY59RPwsAl20bx8wKc9G0DA0Xbx3D4+99PV60b7MMZk4YNJA8k2SoJIFkPXgWJktmjA2eW6vhcs5q0veFq1zG8YPvByFHXOagmNSDxxi8gqWjUvcwt1rH5jEbm8dsEbh+4OwyLuTX5ziXUR+dW0MQMCZ+POJ8m1QUX/BXdx8Rr82v12FyQM9y99zQ50uWgQl+38waI3BmqSokxY+dZADlKh6xMr+WLNklMEy3PpUlPTS7KhhnAnh/+vVn8Po/uastdrjmUA+evOdetGWsrZiEew+exy23PZX6Pq2edNHkTEADcHpqsQJNA9547U6cWqy0zX6RtPzxU9njFgI+mZaHiyaBuq3jBZxdysdNN49y/ECoEQDZk6lpzaXe/VKu57MeaV3D6cVqTEVDyhFLuQFlDjpXevBamdBqus4Uu0NMW07Njq7vi1YWoAMGr+IKdRAgcwDVSa5RD96oRpVz/fKrL8Prr9mJi7eO4Xd/8Bq86frGPXHEylAVTB2aFg4c77f6T2+4Em+/YQ+u3zuFXdNhhvL33nYt3n7D3tBrv/DKS7F1vIAf//+/jQNnV3DL26/DG67ZgfsPz+M//9NjeNdf3w+A9/AZmsj4AWQ/WJzBKzddT3JutA0mG90zXcKDRxfw+KklHJ5dw+xKDU+eXsaTp5fwGJcXvep5jH1qJtMkxm6yaHEJJpmsyNlEgA3MJorxQTkgmYBoTZUsuH6Ap84so2QZeAEHJZNFC7unGZB9+uwKk6XxWXZN0wSL94rL2Dbsni5h51SJM2U1nF9hg8KLt47BMjTMrdXx9afOYaJgour4+PpT58T+AhQGz1QZPCZl++N/ewb/8R8exdc4QPT9AHcfPI9XXr41BJx0TRPHgerN1+3CT7/sInzinqO47/Acbrp0a+h96ocj8EhGY2q24Iv3bRYD0H1bOcDTdTERMGaboRw86r97wzU7AIT77wAGqGxTjxlFuJ6Pt334Hvz2Z7+Lbzw1g0Oza/iXR04hrc4tsx6uS7aN4dxylTt92tA0TbDzwpwjYXBGrBGBcJKwLldcPmNrhUxW1usuqo4fZ/BIopk1B49PTFAlMXiu58PxAhQ5g1dzWUzClnEbm8cKmFutY2ndwf4zy/ieS9ikhKYxae+hGbb/x4sM8K01CTr/2J2H8eTpZXz2OyfFa4vrdUyXbcHMf+gbz+IfHzoBQPYGEoB/4EgchD95egm33nVI/B0EAU4vVsT1RT1kF28dg23omE8xMmHmHnLCIQgBPMkAH+IA76tPnsWxuXWhHmilyBiKTFYmCia2TtiouX7LEqy/vPMQbr3rsJg4ipZw0RQmK8QEpP/O6cUKtk8U8ErO2lOeYKtF0TePn1zMbJATzSrrCODxiasXXDCFubVayJxpI8txyVZfsjK0vY2UAP1UZ5er8APgur1T8PwgJvOPmnsBStC5sn1114+dG6oBV56S1ei51Yho+/r+c3j4eLZoJY/HL9DEbjsumlXHQ93z+dhDMngM4EnCYCTRHNWoulSapuFdL9vXlIn7tddejo/91I3i75su3Yq3vWBPaJDcb3X17kn8yTtfgC+8++WZmMapkoX3ve1a1Fwf77hhD1539Q68+ortqHs+PvPQScEUmFwSoWrxo9l+UYlmo5oWEk22jBft24QHjs7j0w+egG0wIH3rtw7jrR++B//pnx4DIAHeN546h881GMgTYJsQEkyKSQi7aNL7K7XsJivEPD5+agkXbi6LfsOJoolXX7Edpq7hy989g0Mza+KzAOvDM3QNV+2awHTZwt7pEnZNkvtlVUYSTBSwZayA/WeW8djJJfy/r7wEm8oWvvRdlllIoFo1LAEYwKu7Pg6fX8OnH2QD6z/+2jP4wmOn8Sv/8Ajm1+p4+eVhsPbTN12EH74xDPg1TcP//IFr8MEfeT6+9Csvxx+8Iyw73j5RQN31sVRhMjyRdZRy59/Hwb6pa/ieS7bgoz/5Qly/d4o5p3JJ07G5dZi6htdcyQDe5YqDJm3jVbsmYwzet549j9NLVXx9/wz++REGNu56djY20Lj34Hnc/Ie34+HjC9g+UcCOySJn8OqhY8S2I31AOicYPCmj9f2ASXKKcYkmOW3umCxismgKaauY0W7bRVOP9V5VydHP1plEs+7h/FoNW8YKQqL54NF5BAEEwAPY8Tw0y8DOOGfw1mou5iNZc0fPr6Hu+ji1WMFf3nkQY7aB/WeWRa7gwpqDTWVLTBh8/O4j+N0v7sd63WUmK7aB3dMlXLJtDHcnAI0//MoB3HLbARw4y1iihXUHNdfHjRcxSfO9vDdz52QRm8asBgxeEDLSUgeVB2dWoWvsnDw4s4qZlSqe4bEr9x5qHfyQPLJsswzQ6TEL0yWbr38dB2dW8AdfeaphbzHAcgfv5866dzwzk/iZmETTas4EnF6qYPd0CVfvmsSWMRvfatM9lCZWFtadzH14XsDt8cVxaOunAci+u+fvnUYQIBfDpTzK9QNuGKUYkGjS5bhPcGjDInkm3ROik6cUzxMCeJF75MmFdbz8/bfjg//2DA7NruLmP7wd3/dHd4hnNLloNrrfeX6QWRru+n7ERTN5Ry+u1/HuTz2M3/rMY5nYwzxcNOn+P8nHFgCEwUrJNgQ5UGmjz7kfqqsAT9O0N2qa9rSmaQc1TfsvCe8XNE37NH//fk3T9nVzfUY13LV9oigGcwDwuqt34E9yMErpt3r9NTvxuV++GbfwwfyL9m3GdXum8KuvuVwM0A1dg20YQmoISDaJgOTzL5jGlTsnhCStUe2aLKJsS+fR11y1A7MrNfzd/cfxmqu24/l7p/HFx06jZBko2yY2j9m4jmej/fntB/Frn34U//DAcfh+EBvQi5ssN76IBp0TbpsoJJussGXGexgBCWKfPruCCzaXRY/ZRNHCdNnGKy7fir+59yiePreCd774QvG9Gy7chOfvnYJp6PjIj78Qv/qay4XhyS9+8jt4z+efgK6xAfeWcRt3PTMLTQPedN1OvO2GPYL92czNafQEBg8A/uC2p1BxPPzi916KA2dX8KufegQPHJnHyy/bKgAU1S+88lK8/pqdiJaua/ihG/cm9npSFt6rPnAHrvidf8UP/eW9ofWJlmDwuGHRG6/dBU3TYJuaYECOza3jgs1lXLFzAh/+8Rvwoy++ILaca3dP4slTy6EH9WceOoGCySI2vn14HlMlCyfmKziqsCC+H+B3v7QfpxYreOT4IrZPFrFjsoilioNzyzVMl8IAT+anMWbx4MwKFtfrePtH7sEn7jkKQOmTXKtjre7CD+hcM4XJyhOnlsSAevOYjS3jBaUHrzMXzaJp4PRiBb/wvx/CUzx6gwYMJYuZrLAcPCnRXFir477Dc7BNXbBiAAOfc2tMXnnlzkmUbRPnV2t48e9/He/7MpMM3vb4GXzfB+/Ar3/6UdzCX/uzH7sBAHDHM7M4NreG2dUaNo3ZivRaw0rNxRcfO83jG9jA5hWXbcX9h+cxs1LFYQ4sT8yvC9BHrCD1j12zexK2qeOpM8u4ZNsYLt02js1jBdEPubheD2VoRs091H18aHYVF2wu4+rdkzg0u4p7DzLQWLaNRNDZrGquD9vUOQOsY7pki2zNhTUH/+1fnsDH7jwsnGHT6u6D51H3mOvuNw8kA7yoFJqk1Q8msKFUpxer2D1dgq5ruPmyrfjWs+czM3BUnh/gidNLuOFCds48fjJb3EIQBDA0KC6anUk0LUPDNXtY/2WzvM9eVd3zYRlMjkmggJSMmobM+7ru+i0P+JfWHfzDA8fb7h+lOs37tl/GAV7USZMmJ6yEoHM/YP2rv/x3D2NmpYa/vucIbvnyU5hfq2PPphKe5Sy5rrH9kna78/0AP/e3D+L7/ugOzHJ1SKN9F2Xw0nrwPvPQCVQdNul5T4MJnCAIMLNczcVFk8YSNPYAyLxLQ9E0FGn1AKD/hOpac5KmaQaAvwDwOgAnATyoadoXgiDYr3zs5wAsBEFwmaZpPwbg/QDe2a11GtWohqXUQZ9t6vjir7wcAPDzr7gY//TQSVy/ZwqbyzZuulTO/r/jxr3YNV0SN62XXLwZ//prr8z0ez/78ovxumt2CDb0B67fhc1lG5964Dj+w6suxTefnsGjJxbxS6++FK+5cgfOLVdRtk3s3VRCwdSxbaKA3/ncE3jPF57E3ukSXvm8bbjn4Hk+4GK/QQzdE6eW8SufegQrVSfioiklmn95xyFMl5kkV5VWRIuYx5Jt4N/dvA9X7ZrAjRdtwgsvYvvvzdfvxjefnsWe6RJ+8AW7xfd+43XPE2YzN/E+vCAI8P/90PW4/cAMygUDP/zCvZgoWrhy5yRmV2p439uuxWXbJ/Cet1yN//CqS1GpewIwRZ1dKez860/N4M3X7cJvvf4K1BwfV+2awA+9cG8IIHRS+zg7u29LGS+9ZAtuvetwps9H+0FtQ8dK1cGfff1Z3HPovDj/3nL97tgyAOC6PVP4u/uP4/3/egBX7prA6cUqvv7UOfzU9+zDHU/P4PD5Nfz6ay/He7+4H+/5/BMAWL/jwlodB86u4OKtYzhyfg07JosCpD57bgUv5v2fVMREPnRsHr/8dw+j5vq4evckHuEmFbahCwnyzHJNyDQni0yiuVRx8LX95/DLf/ewALCbxxnIoixHQ5FoLq07qHs+7j44i6Pn1/G6q3dgumyhYBrYMmZD01gfTJTBO/9/27vz6LjKM8/j36f2kkqrJUuyNssb3tgdm8VAAANmSUg6JEO2YUIHOtNhJvQkk4WTk9DphiGT6ZBOZyZnOpOEpRNCk4SGdMLiGBOIwYBtbGwMsuVd3iTL2ktSbe/8ca8KybgNBCy5pd/nHJ2qu1TVW1WP6t7n3W5fiie3HGLd7k6+dtW8/IlY1B/nkXPQPZBmSiJCIupNvPTEqwc5s7501Lji6RWFREMBfvipsziluohE1HtsLGT8+I87ae1Msur1dsoKIvx2k9eK/FfL5nDpvKlMLYryN/+6JX9Sc8WCKqaVxjGDW5fN4ZEN3sXrU9lcvub6/FkV3Pv8bi79uz8wkMpy+wcX5D+X0+tLefjl/Xxl+dx8S1FtWZxpJTF2dST57NIZBPxxo+19KZ589SB/9eAGplcU8tPPvI+pRTFyzjuhGv642nqH2N81QE1JjO1tfcyqTDCrMsHjmw+yqrmNkniYa06r4dfr95HK5AiYd2LX57eqRUMBTqsrYfX2wySiYRrKCyiMBklEQwykMkSH/w/DQUoLwvnfh4dfbuXFnd7j/+GpFj54eu2oCiPnnFdWgxVbvK7Y1545jYfWtvoXhvc+r0w2xx9bDrNut9fNbDgOFjeVc97MKfzPx5tZ3DSFWVMT+efvTqbZ25lkX9cAl833KnUumF3Boxv3s+VADw3lBWw91EdpQZiq4tgxe3o45+gZyLC3M0kyleWjZ9ezqbWbV/Z1c0p1ES/tOsKCaSXMqylmX+cAtWVx+gYzdA+kaZhS8OYumu+ie96h7kGmFsXyk6I1H+ylKBYiFgpSVxY/5m+bc47OpDe77dEVdV5lTycXzK4cNQTjncpkc/nf4XAwQCaXzR9bgv4suT2DaR7fdJB5NcXUlsXZsr+HrYe836MlM8p5/WAvN927lo7+FPXlcf7LJbOZW11EbWmc4niY5oO9bGztIhQw6soK2HMkyRn1pfz1b15lzY4j/HbTAf7uY6cztejN49y9SZcCZHKOzv4UqWyOPUeSlBdGmFYa58EX97J2t1dBcHZjGaGA8fyODpbMmMLD61s5s7Es31MgdIxWyu1t/fzXB15mY2s3ty6bzfd+v42Vr7dx84Uz+NqVc3nAf/7hltyWtj5+/sIe1uzoYNH0Mt43vZxtbX28dqCHp5u9Cs0b73mJ1s4kJfEwn1zSSFlhhDU7OkhEQ1wwu4LWzgF2Hu4nEQ3ljymrWw5TFAszqzLBFn/oRE1JjHuf281ZDaXs7kjyv55o5uH1+5hWGmdeTTFTEhFyOcdQJsd9z+9iVXN7/nsb7nK7eV+3P1un10ob8S/vk8nl8rNoh4OB/Azjh/tSrPM/z+Fzj+HYC5j5Mxx7z71+dyfVxTEWN40+/pzsTuTsE4uBFufcDgAz+wVwLTAywbsWuN2//0vgB2Zm7p1WW4kI4J243ri0CfBOBocTDPBmhzy6i9/blYiG8jPigdedZ+nsinw3wuqSGIPpLDee30QsHMy3mjz0uXMpjoVJZXJ8d8VWIqEAa3Z0cN/zuzhvZgXhoHc9uHk1xdSXFVBTEuPp5nZWtxxmMJ2jtCBMZVGUYMCYXlFIzL90xrcff/1NZTzW5TAWNZZx+wfmc8XC6nxS9av/fF5+++ULqqh8PMqty2aP6s56LGbGx95X/6YWq29/5NRRYwWHx0qNNK+mmC8vP4UL/M+rqbKQurI4y+ZVcdtV8wgGjG98YP5xX/9PcXp9Kau+9H4aywsIBIxFjWXc8sDLNJQfe9zltNI49eXxUXEDcGpdKQ+8uJe7f7+VsxvL+NLlpxz3da85fRqrmtv4x2d35CfKmDU1wQ3nNVJdEuVL5CctAAAQwklEQVT//mEH1y9u4N7nd/PstsM0lBfw7DbvO13cVM6dH17IZXc/Q01JjGr/s+wZzFB21BjE4RO+T//4RaaVxKgri/Dyni5uuXgWD63bSygQIBoKUhIPc/fvt3L377d6jysIU1oQZiCd5ab71rKwtphUJsfWQ31UJqL56xuCl+xGggE6+lOc/q0nR73+36/cNmp5eDKW0+reaE0tK4gQDwe56yOncsdvX+NLD23Mb5tSGIERR7uZlYl8JUBr5wCfXNI46vlvuXgWnzqnIX+COHNqgopEhAduOofb/ZPIyxdU8TfXLuTzP1/Pge5B/uKiGZgZy+ZX8YsX9/AXF81gKJ3j0nlTqSqOsfK/XURTRSHVxTG+6JdtuGXv3JlTCAe9Fq+F00r4+r94yfhFcyr5+OIGPvdP65j/jSfyyXFtaZy6sgJ6BzP82Vm1gDdp0R9bDnPz/euYW13EjvZ+lt61ilg4QO9QhqaKgvxJ2neeaOY7TzRTEve6z144p5JZVUXkHDyyYT/LF1Rz4ZxKfvbCHuZ8/bE3xd1bGa4siIUClBVE8l1Uf/TsTurL43zxslO49cENnP23KzC8ryaXc/SnsvkkKJtzXH1aDZfNr+af1uxhyZ0rSURDZHOO/lRmVA+D4IjfhDs/fCpXfO8ZrvjeM4QCxrTSOAWRINvb+/LjW6f5LfvDs+de+4PVOEa3bBZGgpQWRDjSnyIaDlAcC9PWOzhqIp+zG8uYU1XED5/ezg+ffmOsZFE0RO+QN6lE0n9Pc6oS7O5IUlsaz5f3/zy9nfvX7GYwnWUwnfNvs6Szjngk6M/W631nDkgOZRlIZ0lEva7NC6cV53sp3PbwpvzrJ6IhygrD+cmzhq9125VM09Y7RDQUoCIRxfzWRMNo7fQm7YqFvYrCoXSOTM5RGg+TdY5UJudfMiRE70Ca7oE0dWVx//qR3usaRkt7X77M4aAxkB7RRdOMp5vbOf+up445xtv7Dr0T/7qyODcubeKxzQf48i9fyW8PBey4rUgfPbuOX61vZfEdKymKhvItRQHzug/3DmXyCd7RvQUKI8H8LMBTCiMURkP8h/fV87MX9vDIhje3OI88lsXCAa5YUMWD/hjbb1wznxuXNrFudycv7DzCZy9owsz4xJIGPrHE68XyySWN3PnYa9z28CaKYyEe3Tj6NS6ZO5ULZ1dw+2+2sMhPLO/wJx0qiXu/q/c8tyu//9Wn1RAMGDMqClnd0sHqlg6O5ZsfmM/m/T18f+U29nZ6lwo6+rOIh4P82Zm1/OaV/UwtjuaPAcM9GN6pikSU2VOL2N7WT6U/S3VtaZzqkhilBREiwQD3PLeLja1dPPyX5/9JrzFe7ETlUmZ2HbDcOfdZf/nTwBLn3C0j9tns79PqL2/39zl81HPdDNwM0NDQcPbu3W9/+l8RObk459XExY4x+U3PYJrDvUM0VRSyuyNJVXGMeCRIn39h5+HHD6SzdCW9g3n3QJr+oQznzpySv8aR/Nucc8cdl/pvbffGr2UoKXj7tehtPYP0p7JUJCL5WcpGfv+vHeghmcpwdmM5h3oGcY58Qv/M1nZOqfbGQN773C6yObhyYfWo5DOXczy3vYNN+7q55rQaSgvCrNlxhEvnTmV7ex/tfUOcN7OCFVsO8fqBHoJBIxEN8bFF9XQmUzy0tpXywggfPGMaqUyOp5vb+chZtTQf6uXJVw9REAnymfObONA9wOObD+a7a86tLmLW1ASrmtvJ5RyD/iQpmayjJB7m3JlT8l1mewfTdCXT1JcXMJDKsr97gORQlpxznFpbgsPrjlgcC1NdEiOXc7S09xEOBvKJ+fHkcm7UJCXD31025538vnG5gwwdfal8N+Vj2XW4n72dSRY1lucft253J7WlcSoSEVY1t+f/16YURvjRszvpGkhREA5xSnURyxdWs+1QL4PpXL6LdktbH09uOUhRLMx1Z9Wxvb2PRzZ4LXCJWIirT53GvJoivr+yhUTMq+nf1tZLKBDgP503nariGD99bied/Sk+dGYts6cW8aNndzCUzhLya+QjwQBnNpTSM5Bhy4EeLphdQTrrONQzSN9Qhr7BDP1DGeZUF3HFgmp++8oB6svjLJxWwk9We7OLXj6/mrqyOP/wVAvtfYMY3sm3mVEYDRIJepPhVBXHWL6wmopElPue38X29j4G015rYigY4MLZFQymc2w50MNXl88d9f1tb+/jpZ1H2HMkSWvnAMlUhqaKQubVFLNlfw83XTgjXzn0h63tvLizg4AZp9WV0jfkdVM+1DNIZ3+KKYkog+ksvYMZphZFqS7xujKnsjm+csVcVrx2iGe3tTO/poSzGktZ+VobezqSLKgt5rUDvZQXhikriLCquY3SeIRPLGlgSVM5X3hwA13JVL6LmvfnTTgRDgYYSGXoHcp4Yxr9t1boX8+xbyhDQSTIlQtrOH9WBfes3kkm56gqjpFMZdiyv4fewQxZv0U0l3PknJc0zq0u4lDP0Buzmjoveawvi3POjCk8/upBegbS+Uq+roE0QTMioQDJVIa+oSyFEa8yZ1/XgDeRSP55vPPcpbMq+cKy2Xx3xVZe3dfNWY1lfP7iWdz//C7+sLWd4niYjy9uYNuhPnoH0yyYVsKcqgSvH+xlw94uugfSfO6imVQWRcnlHOv3dNKZTLPzcB8d/SlOrS3h9LpSMjnHvs4BqktiPL75ACXxMJ8+dzpb9vewuuUw+7oG8q3COeeNDawsitKVTBEJBaguiRMJepUAzQd7eaW1mxvOa2RTazexcJDrFzfgnOPX6/exsbWLmy6Ywav7e3h++2FCwQBfv3reqN9w5xyPbT5IKpPjQ2fW5n+bD/YMclrdG72BRkqmMuw83M+86mLW7OzgQNcg86cVM5DOcmptCaGAsb29jxkVCcygvW+I7mSapopC+oYyNPs9MWKRIImIl9AOf98HugfZ1tbL/JoS0tkcbb1DTC2KUl9eQCabY3/XIPXlcQbSWXZ3JDnSn8rPKdBQXkBlUZSewTSxUJBw0Gg+1EtXMk06m/P/XP5+wLzPcXgm6OHtUwoj1JcV5I9n6WyO5FCWkoLwG2NoA8aB7gE6+lLEI8H8BFwnEzNb55xbdMxt/x4SvJEWLVrk1q5de0LKLCIiIiIicrI7XoJ3IidZ2QeM7MtU56875j5mFgJKgGO33YqIiIiIiMhxncgE7yVgtpk1mVkEuB549Kh9HgVu8O9fBzyl8XciIiIiIiJ/mhM2YMU5lzGzW4AngCDwE+fcq2b2LWCtc+5R4MfA/WbWAhzBSwJFRERERETkT3BCZyRwzv0O+N1R674x4v4g8NETWQYREREREZHJ4oRe6FxERERERETGjhI8ERERERGRCUIJnoiIiIiIyAShBE9ERERERGSCUIInIiIiIiIyQSjBExERERERmSCU4ImIiIiIiEwQSvBEREREREQmCCV4IiIiIiIiE4QSPBERERERkQlCCZ6IiIiIiMgEoQRPRERERERkglCCJyIiIiIiMkEowRMREREREZkgzDk33mV4R8ysHdg93uU4hgrg8HgXQk5aig85HsWHHI/iQ96KYkSOR/ExMTU65yqPteHfXYJ3sjKztc65ReNdDjk5KT7keBQfcjyKD3krihE5HsXH5KMumiIiIiIiIhOEEjwREREREZEJQgnee+cfx7sAclJTfMjxKD7keBQf8lYUI3I8io9JRmPwREREREREJgi14ImIiIiIiEwQSvDeA2a23MyazazFzL463uWRsWdmPzGzNjPbPGJduZmtMLNt/m2Zv97M7Pt+vLxiZmeNX8llLJhZvZmtMrMtZvaqmX3BX68YEcwsZmYvmtlGPz7+2l/fZGYv+HHwoJlF/PVRf7nF3z59PMsvY8PMgmb2spn9q7+s+BAAzGyXmW0ysw1mttZfp+PLJKYE710ysyDwv4ErgfnAx81s/viWSsbBPcDyo9Z9FVjpnJsNrPSXwYuV2f7fzcAPx6iMMn4ywBedc/OBc4DP+78TihEBGAIucc6dDpwBLDezc4BvA3c752YBncCf+/v/OdDpr7/b308mvi8Ar41YVnzISBc7584YcTkEHV8mMSV4795ioMU5t8M5lwJ+AVw7zmWSMeacewY4ctTqa4F7/fv3Ah8asf4+51kDlJpZzdiUVMaDc+6Ac269f78X7yStFsWIAP733Ocvhv0/B1wC/NJff3R8DMfNL4FLzczGqLgyDsysDrga+H/+sqH4kOPT8WUSU4L37tUCe0cst/rrRKqccwf8+weBKv++YmYS87tLnQm8gGJEfH73uw1AG7AC2A50Oecy/i4jYyAfH/72bmDK2JZYxtj3gC8DOX95CooPeYMDnjSzdWZ2s79Ox5dJLDTeBRCZDJxzzsw0Ze0kZ2YJ4FfArc65npGV6oqRyc05lwXOMLNS4GFg7jgXSU4SZnYN0OacW2dm7x/v8shJaalzbp+ZTQVWmNnrIzfq+DL5qAXv3dsH1I9YrvPXiRwa7vbg37b56xUzk5CZhfGSu585537tr1aMyCjOuS5gFXAuXtep4YrYkTGQjw9/ewnQMcZFlbFzPvBBM9uFNwzkEuDvUXyIzzm3z79tw6sgWoyOL5OaErx37yVgtj+bVQS4Hnh0nMskJ4dHgRv8+zcAj4xY/x/9mazOAbpHdKOQCcgf//Jj4DXn3HdHbFKMCGZW6bfcYWZx4DK8cZqrgOv83Y6Oj+G4uQ54yumithOWc+5rzrk659x0vHOMp5xzn0TxIYCZFZpZ0fB94HJgMzq+TGq60Pl7wMyuwusfHwR+4py7Y5yLJGPMzB4A3g9UAIeAbwL/Avwz0ADsBj7mnDvin+z/AG/WzSTwGefc2vEot4wNM1sKPAts4o0xNLfhjcNTjExyZnYa3iQIQbyK1392zn3LzGbgtdiUAy8Dn3LODZlZDLgfbyznEeB659yO8Sm9jCW/i+aXnHPXKD4EwI+Dh/3FEPBz59wdZjYFHV8mLSV4IiIiIiIiE4S6aIqIiIiIiEwQSvBEREREREQmCCV4IiIiIiIiE4QSPBERERERkQlCCZ6IiIiIiMgEoQRPREQmLTPLmtkGM9toZuvN7Ly32L/UzP7ybTzv02a26L0rqYiIyNujBE9ERCazAefcGc6504GvAf/jLfYvBd4ywRMRERkvSvBEREQ8xUAngJklzGyl36q3ycyu9fe5C5jpt/p9x9/3K/4+G83srhHP91Eze9HMtprZBWP7VkREZLIKjXcBRERExlHczDYAMaAGuMRfPwh82DnXY2YVwBozexT4KrDQOXcGgJldCVwLLHHOJc2sfMRzh5xzi83sKuCbwLIxek8iIjKJKcETEZHJbGBEsnYucJ+ZLQQMuNPMLgRyQC1QdYzHLwN+6pxLAjjnjozY9mv/dh0w/cQUX0REZDQleCIiIoBz7nm/ta4SuMq/Pds5lzazXXitfO/EkH+bRcdbEREZIxqDJyIiApjZXCAIdAAlQJuf3F0MNPq79QJFIx62AviMmRX4zzGyi6aIiMiYU42iiIhMZsNj8MDrlnmDcy5rZj8DfmNmm4C1wOsAzrkOM1ttZpuBx5xz/93MzgDWmlkK+B1w2zi8DxEREQDMOTfeZRAREREREZH3gLpoioiIiIiITBBK8ERERERERCYIJXgiIiIiIiIThBI8ERERERGRCUIJnoiIiIiIyAShBE9ERERERGSCUIInIiIiIiIyQSjBExERERERmSD+Pwk6zdaEAxDOAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { + "needs_background": "light", "tags": [] }, "output_type": "display_data" @@ -1102,55 +938,55 @@ " ## TRAINING\n", " \n", " # Set our model to training mode\n", - " model.train() \n", - " # Tracking variables\n", - " tr_loss = 0\n", - " nb_tr_examples, nb_tr_steps = 0, 0\n", + " model.train() \n", + " # Tracking variables\n", + " tr_loss = 0\n", + " nb_tr_examples, nb_tr_steps = 0, 0\n", " # Train the data for one epoch\n", - " for step, batch in enumerate(train_dataloader):\n", + " for step, batch in enumerate(train_dataloader):\n", " # Add batch to GPU\n", - " batch = tuple(t.to(device) for t in batch)\n", - " # Unpack the inputs from our dataloader\n", - " b_input_ids, b_input_mask, b_labels = batch\n", - " # Clear out the gradients (by default they accumulate)\n", - " optimizer.zero_grad()\n", - " # Forward pass\n", - " loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n", - " train_loss_set.append(loss.item()) \n", - " # Backward pass\n", - " loss.backward()\n", - " # Update parameters and take a step using the computed gradient\n", - " optimizer.step()\n", - " # Update tracking variables\n", - " tr_loss += loss.item()\n", - " nb_tr_examples += b_input_ids.size(0)\n", - " nb_tr_steps += 1\n", - " print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n", + " batch = tuple(t.to(device) for t in batch)\n", + " # Unpack the inputs from our dataloader\n", + " b_input_ids, b_input_mask, b_labels = batch\n", + " # Clear out the gradients (by default they accumulate)\n", + " optimizer.zero_grad()\n", + " # Forward pass\n", + " loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n", + " train_loss_set.append(loss.item()) \n", + " # Backward pass\n", + " loss.backward()\n", + " # Update parameters and take a step using the computed gradient\n", + " optimizer.step()\n", + " # Update tracking variables\n", + " tr_loss += loss.item()\n", + " nb_tr_examples += b_input_ids.size(0)\n", + " nb_tr_steps += 1\n", + " print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n", " \n", " ## VALIDATION\n", "\n", - " # Put model in evaluation mode\n", - " model.eval()\n", - " # Tracking variables \n", - " eval_loss, eval_accuracy = 0, 0\n", - " nb_eval_steps, nb_eval_examples = 0, 0\n", - " # Evaluate data for one epoch\n", - " for batch in validation_dataloader:\n", + " # Put model in evaluation mode\n", + " model.eval()\n", + " # Tracking variables \n", + " eval_loss, eval_accuracy = 0, 0\n", + " nb_eval_steps, nb_eval_examples = 0, 0\n", + " # Evaluate data for one epoch\n", + " for batch in validation_dataloader:\n", " # Add batch to GPU\n", - " batch = tuple(t.to(device) for t in batch)\n", - " # Unpack the inputs from our dataloader\n", - " b_input_ids, b_input_mask, b_labels = batch\n", - " # Telling the model not to compute or store gradients, saving memory and speeding up validation\n", - " with torch.no_grad():\n", - " # Forward pass, calculate logit predictions\n", - " logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask) \n", - " # Move logits and labels to CPU\n", - " logits = logits.detach().cpu().numpy()\n", - " label_ids = b_labels.to('cpu').numpy()\n", - " tmp_eval_accuracy = flat_accuracy(logits, label_ids) \n", - " eval_accuracy += tmp_eval_accuracy\n", - " nb_eval_steps += 1\n", - " print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n", + " batch = tuple(t.to(device) for t in batch)\n", + " # Unpack the inputs from our dataloader\n", + " b_input_ids, b_input_mask, b_labels = batch\n", + " # Telling the model not to compute or store gradients, saving memory and speeding up validation\n", + " with torch.no_grad():\n", + " # Forward pass, calculate logit predictions\n", + " logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask) \n", + " # Move logits and labels to CPU\n", + " logits = logits.detach().cpu().numpy()\n", + " label_ids = b_labels.to('cpu').numpy()\n", + " tmp_eval_accuracy = flat_accuracy(logits, label_ids) \n", + " eval_accuracy += tmp_eval_accuracy\n", + " nb_eval_steps += 1\n", + " print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n", "\n", "# plot training performance\n", "plt.figure(figsize=(15,8))\n", @@ -1160,24 +996,13 @@ "plt.plot(train_loss_set)\n", "plt.show()" ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "WYOQdgILrkiX" - }, - "outputs": [], - "source": [] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], - "name": "BERT_ATIS.ipynb", + "name": "03_BERT_ATIS_Binary.ipynb", "provenance": [] }, "kernelspec": { @@ -1195,7 +1020,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.12" + "version": "3.6.13" } }, "nbformat": 4, diff --git a/Ch6/04_CRF_SNIPS_slots.ipynb b/Ch6/04_CRF_SNIPS_slots.ipynb index 4bf2698..a24f6e3 100755 --- a/Ch6/04_CRF_SNIPS_slots.ipynb +++ b/Ch6/04_CRF_SNIPS_slots.ipynb @@ -7,14 +7,27 @@ "ExecuteTime": { "end_time": "2020-01-28T19:01:10.723482Z", "start_time": "2020-01-28T19:01:08.616836Z" - } + }, + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "KQRRQ3yzPpUw", + "outputId": "dc65b111-045b-42a9-c3fb-2664e88652a5" }, "outputs": [ { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ - "Using TensorFlow backend.\n" + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] /root/nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "Requirement already satisfied: sklearn_crfsuite in /usr/local/lib/python3.7/dist-packages (0.3.6)\n", + "Requirement already satisfied: python-crfsuite>=0.8.3 in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (0.9.7)\n", + "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (1.15.0)\n", + "Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (0.8.9)\n", + "Requirement already satisfied: tqdm>=2.0 in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (4.41.1)\n" ] } ], @@ -22,11 +35,15 @@ "import os\n", "import json\n", "import string\n", + "import wget\n", + "from zipfile import ZipFile\n", "\n", "import numpy as np\n", - "\n", + "import nltk\n", + "nltk.download('averaged_perceptron_tagger')\n", "from nltk.tag import pos_tag\n", "\n", + "!pip install sklearn_crfsuite\n", "from sklearn_crfsuite import CRF, metrics\n", "from sklearn.metrics import make_scorer,confusion_matrix\n", "from sklearn.metrics import f1_score,classification_report\n", @@ -44,12 +61,70 @@ "ExecuteTime": { "end_time": "2020-01-28T19:01:10.785317Z", "start_time": "2020-01-28T19:01:10.725237Z" - } + }, + "colab": { + "base_uri": "https://localhost:8080/", + "height": 106, + "resources": { + "http://localhost:8080/nbextensions/google.colab/files.js": { + "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK", + "headers": [ + [ + "content-type", + "application/javascript" + ] + ], + "ok": true, + "status": 200, + "status_text": "" + } + } + }, + "id": "rpHGh0dyPpUz", + "outputId": "9b080802-cd27-4946-8f6a-e5231ee6e9b5" }, - "outputs": [], + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " Upload widget is only available when the cell has been executed in the\n", + " current browser session. Please rerun this cell to enable.\n", + " \n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saving train_PlayMusic_full.json to train_PlayMusic_full.json\n", + "Saving validate_PlayMusic.json to validate_PlayMusic.json\n" + ] + } + ], "source": [ - "train_loc = \"Data/snips/train_PlayMusic_full.json\"\n", - "test_loc = \"Data/snips/validate_PlayMusic.json\"\n", + "try:\n", + " from google.colab import files\n", + "# upload \"train_PlayMusic_full.json\" and \"validate_PlayMusic.json\" here\n", + " uploaded = files.upload()\n", + " train_loc = \"train_PlayMusic_full.json\"\n", + " test_loc = \"validate_PlayMusic.json\"\n", + " \n", + "except ModuleNotFoundError:\n", + " train_loc = \"Data/snips/train_PlayMusic_full.json\"\n", + " test_loc = \"Data/snips/validate_PlayMusic.json\"\n", "\n", "train_file = json.load(open(train_loc, encoding= \"iso-8859-2\"))\n", "test_file = json.load(open(test_loc, encoding= \"iso-8859-2\"))" @@ -62,7 +137,8 @@ "ExecuteTime": { "end_time": "2020-01-28T19:01:10.791023Z", "start_time": "2020-01-28T19:01:10.786037Z" - } + }, + "id": "LwUcnYOwPpU0" }, "outputs": [], "source": [ @@ -77,7 +153,8 @@ "ExecuteTime": { "end_time": "2020-01-28T19:01:10.798005Z", "start_time": "2020-01-28T19:01:10.792054Z" - } + }, + "id": "crYoV0k9PpU0" }, "outputs": [], "source": [ @@ -109,7 +186,12 @@ "ExecuteTime": { "end_time": "2020-01-28T19:01:11.033375Z", "start_time": "2020-01-28T19:01:10.799002Z" - } + }, + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "2n3GrvhXPpU1", + "outputId": "cecb04b9-5666-44de-906e-1f49eb5ac448" }, "outputs": [ { @@ -2226,18 +2308,73 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:11.037401Z", "start_time": "2020-01-28T19:01:11.034387Z" - } + }, + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "6lkzF_a3PpU1", + "outputId": "ae9b6255-109a-44a7-edd3-a9c4af53b061" }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--2021-06-09 11:23:24-- http://nlp.stanford.edu/data/glove.6B.zip\n", + "Resolving nlp.stanford.edu (nlp.stanford.edu)... 171.64.67.140\n", + "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:80... connected.\n", + "HTTP request sent, awaiting response... 302 Found\n", + "Location: https://nlp.stanford.edu/data/glove.6B.zip [following]\n", + "--2021-06-09 11:23:24-- https://nlp.stanford.edu/data/glove.6B.zip\n", + "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:443... connected.\n", + "HTTP request sent, awaiting response... 301 Moved Permanently\n", + "Location: http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip [following]\n", + "--2021-06-09 11:23:25-- http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip\n", + "Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 171.64.64.22\n", + "Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|171.64.64.22|:80... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 862182613 (822M) [application/zip]\n", + "Saving to: ‘glove.6B/glove.6B.zip’\n", + "\n", + "glove.6B.zip 100%[===================>] 822.24M 5.12MB/s in 2m 42s \n", + "\n", + "2021-06-09 11:26:07 (5.08 MB/s) - ‘glove.6B/glove.6B.zip’ saved [862182613/862182613]\n", + "\n", + "Archive: glove.6B/glove.6B.zip\n", + " inflating: glove.6B/glove.6B.50d.txt \n", + " inflating: glove.6B/glove.6B.100d.txt \n", + " inflating: glove.6B/glove.6B.200d.txt \n", + " inflating: glove.6B/glove.6B.300d.txt \n" + ] + } + ], "source": [ - "BASE_DIR = 'Data'\n", - "GLOVE_DIR = os.path.join(BASE_DIR, 'glove.6B')\n", + "try :\n", + " from google.colab import files\n", + " !wget -P glove.6B http://nlp.stanford.edu/data/glove.6B.zip\n", + " !unzip glove.6B/glove.6B.zip -d glove.6B\n", + " BASE_DIR='.'\n", + "except ModuleNotFoundError :\n", + " if not os.path.exists(os.getcwd()+'\\\\Data\\\\glove.6B'):\n", + " os.mkdir(os.getcwd()+'\\\\Data\\\\glove.6B')\n", "\n", + " url='http://nlp.stanford.edu/data/glove.6B.zip' \n", + " path=os.getcwd()+'\\Data' \n", + " wget.download(url,path) \n", + "\n", + " temp=path+'\\glove.6B.zip' \n", + " file = ZipFile(temp) \n", + " file.extractall(path+'\\glove.6B') \n", + " file.close() \n", + " \n", + " BASE_DIR = 'Data' \n", + " \n", + "GLOVE_DIR = os.path.join(BASE_DIR, 'glove.6B')\n", "MAX_SEQUENCE_LENGTH = 300\n", "MAX_NUM_WORDS = 20000 \n", "EMBEDDING_DIM = 100 \n", @@ -2246,12 +2383,17 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:22.707203Z", "start_time": "2020-01-28T19:01:11.039359Z" - } + }, + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "IEGeeSVEPpU2", + "outputId": "14ba9578-6b14-43e9-9d2c-6a5213d59a88" }, "outputs": [ { @@ -2259,7 +2401,7 @@ "output_type": "stream", "text": [ "Preparing embedding matrix.\n", - "Found 400001 word vectors in Glove embeddings.\n" + "Found 400000 word vectors in Glove embeddings.\n" ] } ], @@ -2289,22 +2431,33 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 8, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:22.719134Z", "start_time": "2020-01-28T19:01:22.709201Z" - } + }, + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "id": "MSwvgNxaPpU3", + "outputId": "163ad86a-d238-4b70-a17c-3ca2a26d833e" }, "outputs": [ { "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + }, "text/plain": [ "'I need to hear the song Aspro Mavro from Bill Szymczyk on Youtube'" ] }, - "execution_count": 9, - "metadata": {}, + "execution_count": 8, + "metadata": { + "tags": [] + }, "output_type": "execute_result" } ], @@ -2317,12 +2470,17 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 9, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:22.773986Z", "start_time": "2020-01-28T19:01:22.720185Z" - } + }, + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "jNhOyHPbPpU3", + "outputId": "1c559d69-f92c-4933-a4c8-7d960f509fc2" }, "outputs": [ { @@ -2344,12 +2502,13 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 10, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:22.785954Z", "start_time": "2020-01-28T19:01:22.775018Z" - } + }, + "id": "R32CcFGaPpU3" }, "outputs": [], "source": [ @@ -2422,12 +2581,13 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 11, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:22.792969Z", "start_time": "2020-01-28T19:01:22.786951Z" - } + }, + "id": "UWvHwHgjPpU4" }, "outputs": [], "source": [ @@ -2443,12 +2603,13 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 12, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:22.799982Z", "start_time": "2020-01-28T19:01:22.793978Z" - } + }, + "id": "u30ziBm0PpU5" }, "outputs": [], "source": [ @@ -2469,12 +2630,13 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 13, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:22.807948Z", "start_time": "2020-01-28T19:01:22.800914Z" - } + }, + "id": "Kn-9DabIPpU5" }, "outputs": [], "source": [ @@ -2502,12 +2664,13 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 14, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:22.813930Z", "start_time": "2020-01-28T19:01:22.808893Z" - } + }, + "id": "zkjAIvckPpU5" }, "outputs": [], "source": [ @@ -2523,19 +2686,43 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 15, "metadata": { "ExecuteTime": { "end_time": "2020-01-28T19:01:34.092725Z", "start_time": "2020-01-28T19:01:22.814878Z" - } + }, + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "duWlQXq8PpU6", + "outputId": "76ab9975-37fc-4573-db35-fcda9b7fd642", + "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Training a Sequence classification model with CRF\n", + "Training a Sequence classification model with CRF\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1515: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true nor predicted samples. Use `zero_division` parameter to control this behavior.\n", + " average, \"true nor predicted\", 'F-score is', len(true_sum)\n", + "/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n", + " _warn_prf(average, modifier, msg_start, len(result))\n", + "/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n", + " _warn_prf(average, modifier, msg_start, len(result))\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ "0.8560889758746073\n", " precision recall f1-score support\n", "\n", @@ -2633,20 +2820,6 @@ " music_item-2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 1 3\n", "Done with sequence model\n" ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/home/samyak/OpenSource/practical-nlp/env/lib64/python3.8/site-packages/sklearn/metrics/_classification.py:1464: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true nor predicted samples. Use `zero_division` parameter to control this behavior.\n", - " _warn_prf(\n", - "/home/samyak/OpenSource/practical-nlp/env/lib64/python3.8/site-packages/sklearn/utils/validation.py:68: FutureWarning: Pass labels=['O', 'year-1', 'genre-1', 'genre-2', 'genre-3', 'genre-4', 'genre-5', 'genre-6', 'service-1', 'service-2', 'playlist-1', 'playlist-2', 'playlist-3', 'playlist-4', 'playlist-5', 'playlist-6', 'album-1', 'album-2', 'album-3', 'album-4', 'album-5', 'album-6', 'album-7', 'album-8', 'sort-1', 'sort-2', 'track-1', 'track-2', 'track-3', 'track-4', 'track-5', 'track-6', 'track-7', 'track-8', 'artist-1', 'artist-2', 'artist-3', 'artist-4', 'artist-5', 'artist-6', 'music_item-1', 'music_item-2'] as keyword args. From version 0.25 passing these as positional arguments will result in an error\n", - " warnings.warn(\"Pass {} as keyword args. From version 0.25 \"\n", - "/home/samyak/OpenSource/practical-nlp/env/lib64/python3.8/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n", - " _warn_prf(average, modifier, msg_start, len(result))\n", - "/home/samyak/OpenSource/practical-nlp/env/lib64/python3.8/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n", - " _warn_prf(average, modifier, msg_start, len(result))\n" - ] } ], "source": [ @@ -2656,16 +2829,14 @@ "train_seq(feats, labels, devfeats, devlabels)\n", "print(\"Done with sequence model\")" ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { + "colab": { + "collapsed_sections": [], + "name": "04_CRF_SNIPS_slots.ipynb", + "provenance": [] + }, "kernelspec": { "display_name": "Python 3", "language": "python", @@ -2681,7 +2852,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.12" + "version": "3.6.13" }, "toc": { "base_numbering": 1, @@ -2727,5 +2898,5 @@ } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 1 } diff --git a/Ch6/05_BERT_SNIPS.ipynb b/Ch6/05_BERT_SNIPS.ipynb index 8fc053c..5426ef6 100644 --- a/Ch6/05_BERT_SNIPS.ipynb +++ b/Ch6/05_BERT_SNIPS.ipynb @@ -3,7 +3,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "LqXsulG-foOX" }, "source": [ @@ -14,7 +13,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "CMqpe5rnf1op" }, "source": [ @@ -24,52 +22,79 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 1, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 459 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "Gety72pIhW7j", - "outputId": "d78daa3f-170b-4cff-92d7-43786488034c" + "outputId": "7b228da8-4da2-4919-eccd-3f00eb974cbb" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Requirement already satisfied: pytorch-pretrained-bert==0.4.0 in /usr/local/lib/python3.6/dist-packages (0.4.0)\n", - "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.13.13)\n", - "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.18.4)\n", - "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (4.41.1)\n", - "Requirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.5.0+cu101)\n", - "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert==0.4.0) (2.23.0)\n", - "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert==0.4.0) (0.10.0)\n", - "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert==0.4.0) (0.3.3)\n", - "Requirement already satisfied: botocore<1.17.0,>=1.16.13 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert==0.4.0) (1.16.13)\n", - "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch>=0.4.1->pytorch-pretrained-bert==0.4.0) (0.16.0)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (2020.4.5.1)\n", - "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (2.9)\n", - "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (3.0.4)\n", - "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (1.24.3)\n", - "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.13->boto3->pytorch-pretrained-bert==0.4.0) (0.15.2)\n", - "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.13->boto3->pytorch-pretrained-bert==0.4.0) (2.8.1)\n", - "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.17.0,>=1.16.13->boto3->pytorch-pretrained-bert==0.4.0) (1.12.0)\n", - "Requirement already satisfied: seqeval==0.0.12 in /usr/local/lib/python3.6/dist-packages (0.0.12)\n", - "Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.6/dist-packages (from seqeval==0.0.12) (1.18.4)\n", - "Requirement already satisfied: Keras>=2.2.4 in /usr/local/lib/python3.6/dist-packages (from seqeval==0.0.12) (2.3.1)\n", - "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.0.8)\n", - "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (3.13)\n", - "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (2.10.0)\n", - "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.1.2)\n", - "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.4.1)\n", - "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.12.0)\n" + "TensorFlow 1.x selected.\n", + "Collecting pytorch-pretrained-bert==0.4.0\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/95/68/84de54aea460eb5b2e90bf47a429aacc1ce97ff052ec40874ea38ae2331d/pytorch_pretrained_bert-0.4.0-py3-none-any.whl (45kB)\n", + "\u001b[K |████████████████████████████████| 51kB 2.7MB/s \n", + "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert==0.4.0) (2.23.0)\n", + "Collecting boto3\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/7c/e1/1b164502f455035def771ec7a31f705351b7f953695d57ce26219aaf21a9/boto3-1.17.90-py2.py3-none-any.whl (131kB)\n", + "\u001b[K |████████████████████████████████| 133kB 7.1MB/s \n", + "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert==0.4.0) (4.41.1)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.19.5)\n", + "Requirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from pytorch-pretrained-bert==0.4.0) (1.8.1+cu101)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (2020.12.5)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (3.0.4)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (2.10)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pytorch-pretrained-bert==0.4.0) (1.24.3)\n", + "Collecting s3transfer<0.5.0,>=0.4.0\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/63/d0/693477c688348654ddc21dcdce0817653a294aa43f41771084c25e7ff9c7/s3transfer-0.4.2-py2.py3-none-any.whl (79kB)\n", + "\u001b[K |████████████████████████████████| 81kB 10.8MB/s \n", + "\u001b[?25hCollecting botocore<1.21.0,>=1.20.90\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/4a/ac/617d3ac25ea905279deb06edd82d6c19ca272006d6dcf232b837b75c3dde/botocore-1.20.90-py2.py3-none-any.whl (7.6MB)\n", + "\u001b[K |████████████████████████████████| 7.6MB 46.2MB/s \n", + "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n", + " Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n", + "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=0.4.1->pytorch-pretrained-bert==0.4.0) (3.7.4.3)\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.21.0,>=1.20.90->boto3->pytorch-pretrained-bert==0.4.0) (2.8.1)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.21.0,>=1.20.90->boto3->pytorch-pretrained-bert==0.4.0) (1.15.0)\n", + "\u001b[31mERROR: botocore 1.20.90 has requirement urllib3<1.27,>=1.25.4, but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n", + "Installing collected packages: jmespath, botocore, s3transfer, boto3, pytorch-pretrained-bert\n", + "Successfully installed boto3-1.17.90 botocore-1.20.90 jmespath-0.10.0 pytorch-pretrained-bert-0.4.0 s3transfer-0.4.2\n", + "Collecting seqeval==0.0.12\n", + " Downloading https://files.pythonhosted.org/packages/34/91/068aca8d60ce56dd9ba4506850e876aba5e66a6f2f29aa223224b50df0de/seqeval-0.0.12.tar.gz\n", + "Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from seqeval==0.0.12) (1.19.5)\n", + "Requirement already satisfied: Keras>=2.2.4 in /tensorflow-1.15.2/python3.7 (from seqeval==0.0.12) (2.3.1)\n", + "Requirement already satisfied: keras-applications>=1.0.6 in /tensorflow-1.15.2/python3.7 (from Keras>=2.2.4->seqeval==0.0.12) (1.0.8)\n", + "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.4.1)\n", + "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.15.0)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (3.13)\n", + "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (1.1.2)\n", + "Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from Keras>=2.2.4->seqeval==0.0.12) (3.1.0)\n", + "Requirement already satisfied: cached-property; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from h5py->Keras>=2.2.4->seqeval==0.0.12) (1.5.2)\n", + "Building wheels for collected packages: seqeval\n", + " Building wheel for seqeval (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for seqeval: filename=seqeval-0.0.12-cp37-none-any.whl size=7436 sha256=cb9b93bd09bfd6e7304259f8032d02ae36c7e7c5dd3e23ece354e0360c611bb9\n", + " Stored in directory: /root/.cache/pip/wheels/4f/32/0a/df3b340a82583566975377d65e724895b3fad101a3fb729f68\n", + "Successfully built seqeval\n", + "Installing collected packages: seqeval\n", + "Successfully installed seqeval-0.0.12\n", + "Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Using TensorFlow backend.\n" ] } ], "source": [ - "%tensorflow_version 1.x \n", + "% tensorflow_version 1.x \n", "\n", "#Installing required packages \n", "!pip install pytorch-pretrained-bert==0.4.0\n", @@ -94,13 +119,12 @@ "from sklearn.model_selection import train_test_split\n", "\n", "#importing packages to calculate the f1_score of our model\n", - "from seqeval.metrics import f1_score\n" + "from seqeval.metrics import f1_score" ] }, { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "CuXoMTpfhcLu" }, "source": [ @@ -109,14 +133,14 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 108, + "height": 106, "resources": { "http://localhost:8080/nbextensions/google.colab/files.js": { - "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=", + "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK", "headers": [ [ "content-type", @@ -129,17 +153,17 @@ } } }, - "colab_type": "code", "id": "QrdCfa9D7sDx", - "outputId": "d212ec70-9fbe-4a77-a9f1-1234acdc3ec8" + "outputId": "2c61bfc8-068f-49fd-ae99-c292c864da12" }, "outputs": [ { "data": { "text/html": [ "\n", - " \n", - " \n", + " \n", + " \n", " Upload widget is only available when the cell has been executed in the\n", " current browser session. Please rerun this cell to enable.\n", " \n", @@ -158,20 +182,25 @@ "name": "stdout", "output_type": "stream", "text": [ - "Saving validate_PlayMusic.json to validate_PlayMusic.json\n", - "Saving train_PlayMusic_full.json to train_PlayMusic_full (1).json\n" + "Saving train_PlayMusic_full.json to train_PlayMusic_full.json\n", + "Saving validate_PlayMusic.json to validate_PlayMusic.json\n" ] } ], "source": [ - "from google.colab import files\n", - "uploaded = files.upload()" + "try:\n", + " from google.colab import files\n", + " uploaded = files.upload()\n", + " train_loc = \"train_PlayMusic_full.json\"\n", + " test_loc = \"validate_PlayMusic.json\"\n", + "except ModuleNotFoundError:\n", + " train_loc = \"Data/snips/train_PlayMusic_full.json\"\n", + " test_loc = \"Data/snips/validate_PlayMusic.json\"" ] }, { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "9nJiVB2dhfP4" }, "source": [ @@ -180,17 +209,12 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 3, "metadata": { - "colab": {}, - "colab_type": "code", "id": "2ejxRP-V7xE1" }, "outputs": [], "source": [ - "train_loc = \"train_PlayMusic_full.json\"\n", - "test_loc = \"validate_PlayMusic.json\"\n", - "\n", "train_file = json.load(open(train_loc, encoding= \"iso-8859-2\"))\n", "test_file = json.load(open(test_loc, encoding= \"iso-8859-2\"))" ] @@ -198,7 +222,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "ImDVF-qOhp-V" }, "source": [ @@ -207,10 +230,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 4, "metadata": { - "colab": {}, - "colab_type": "code", "id": "fJ8HlEyr8bs5" }, "outputs": [], @@ -222,7 +243,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "8mYinws6hjUj" }, "source": [ @@ -231,10 +251,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 5, "metadata": { - "colab": {}, - "colab_type": "code", "id": "I5fmDAoZ8dx6" }, "outputs": [], @@ -262,15 +280,14 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, - "colab_type": "code", "id": "UkvdSDU08ilm", - "outputId": "10a52b43-6ff2-44c1-aa9b-415c01c37b7e" + "outputId": "8c26ef72-c04f-4401-b755-797c6f83adad" }, "outputs": [ { @@ -337,7 +354,7 @@ "4 [Play, Magic, Sam, from, the, thirties] [O, artist-1, artist-2, O, O, year-1]" ] }, - "execution_count": 25, + "execution_count": 6, "metadata": { "tags": [] }, @@ -356,7 +373,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "eKiCnohqhzf2" }, "source": [ @@ -365,33 +381,31 @@ }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 7, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 187 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "ybF5FSMP8kn8", - "outputId": "df9a5e6c-0da2-41cc-8c90-c375233abe0a" + "outputId": "0a1d8143-0e65-48f5-d3f3-21d4f2857c72" }, "outputs": [ { "data": { "text/plain": [ - "['O',\n", - " 'service-2',\n", - " 'track-1',\n", - " 'track-8',\n", + "['genre-4',\n", + " 'track-7',\n", + " 'album-2',\n", + " 'playlist-4',\n", + " 'artist-3',\n", " 'artist-5',\n", - " 'artist-1',\n", - " 'album-1',\n", - " 'album-5',\n", - " 'artist-2',\n", - " 'year-1']" + " 'service-1',\n", + " 'artist-4',\n", + " 'playlist-3',\n", + " 'track-4']" ] }, - "execution_count": 46, + "execution_count": 7, "metadata": { "tags": [] }, @@ -404,27 +418,23 @@ "\n", "unique_labels=[]\n", "for i in label:\n", - " unique_labels += i\n", - " \n", - "labels = unique_labels # all the labels\n", + " unique_labels += i\n", "\n", + "labels = unique_labels # all the labels\n", "unique_labels = set(unique_labels)#set of all the unique labels\n", "\n", - "list(unique_labels)[:10]\n", - "\n" + "list(unique_labels)[:10]" ] }, { "cell_type": "code", - "execution_count": 47, + "execution_count": 8, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 255 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "vYjw7Hzgk0Q4", - "outputId": "8f569743-7fd7-4ca4-b686-285cef421716" + "outputId": "ec556f3e-db9a-49bd-9fee-7492060987f9" }, "outputs": [ { @@ -446,7 +456,7 @@ " ['play', 'Yo', 'Ho', 'from', 'the', 'new', 'york', 'pops', 'on', 'Youtube']]" ] }, - "execution_count": 47, + "execution_count": 8, "metadata": { "tags": [] }, @@ -460,7 +470,6 @@ { "cell_type": "markdown", "metadata": { - "colab_type": "text", "id": "x3sMXj_ukmOh" }, "source": [ @@ -469,15 +478,12 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 9, "metadata": { - "colab": {}, - "colab_type": "code", "id": "EjoeGhClj78E" }, "outputs": [], "source": [ - "\n", "import re\n", "def untokenize(words):\n", " \"\"\"\n", @@ -496,29 +502,31 @@ " step6 = step5.replace(\" ` \", \" '\")\n", " return step6.strip()\n", "\n", - "sentences_untokenized = [untokenize(i) for i in sentence]\n" + "sentences_untokenized = [untokenize(i) for i in sentence]" ] }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 34 + "height": 35 }, - "colab_type": "code", "id": "rNxrLnae2gef", - "outputId": "5788c12c-14b0-49b8-eb4b-7fbc76f95fab" + "outputId": "12445b63-82ff-44f8-a017-97ad343c0a36" }, "outputs": [ { "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + }, "text/plain": [ "'I need to hear the song Aspro Mavro from Bill Szymczyk on Youtube'" ] }, - "execution_count": 49, + "execution_count": 10, "metadata": { "tags": [] }, @@ -531,17 +539,22 @@ }, { "cell_type": "code", - "execution_count": 50, + "execution_count": 11, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 54 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "WXieM-1p-LGC", - "outputId": "041fe2eb-9bf1-4c8c-a22b-9bd6a2a118a3" + "outputId": "8e772414-86d7-49f7-c115-46dc2cb91c58" }, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 231508/231508 [00:00<00:00, 305110.74B/s]\n" + ] + }, { "name": "stdout", "output_type": "stream", @@ -569,15 +582,13 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 12, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "l5o3olMNDNTO", - "outputId": "f9c75f04-c54a-4264-c5ab-665fa850b603" + "outputId": "b3640c5c-4552-478a-dece-cf4ea8d413f1" }, "outputs": [ { @@ -586,7 +597,7 @@ "(15292,)" ] }, - "execution_count": 51, + "execution_count": 12, "metadata": { "tags": [] }, @@ -599,33 +610,31 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 13, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 187 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "iTMARnfdDSFN", - "outputId": "1ec2751e-19d4-4d42-e401-2969cc39afd0" + "outputId": "f389d9d4-036e-4bbb-b485-ccdee82761d9" }, "outputs": [ { "data": { "text/plain": [ - "['O',\n", - " 'service-2',\n", - " 'track-1',\n", - " 'track-8',\n", + "['genre-4',\n", + " 'track-7',\n", + " 'album-2',\n", + " 'playlist-4',\n", + " 'artist-3',\n", " 'artist-5',\n", - " 'artist-1',\n", - " 'album-1',\n", - " 'album-5',\n", - " 'artist-2',\n", - " 'year-1']" + " 'service-1',\n", + " 'artist-4',\n", + " 'playlist-3',\n", + " 'track-4']" ] }, - "execution_count": 52, + "execution_count": 13, "metadata": { "tags": [] }, @@ -641,10 +650,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 14, "metadata": { - "colab": {}, - "colab_type": "code", "id": "H8OoO45gDg8b" }, "outputs": [], @@ -660,10 +667,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 15, "metadata": { - "colab": {}, - "colab_type": "code", "id": "fwznylB1DiiH" }, "outputs": [], @@ -677,10 +682,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 16, "metadata": { - "colab": {}, - "colab_type": "code", "id": "gosRXgG-D50X" }, "outputs": [], @@ -694,10 +697,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 17, "metadata": { - "colab": {}, - "colab_type": "code", "id": "jPLqDFjQFTcq" }, "outputs": [], @@ -714,10 +715,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 18, "metadata": { - "colab": {}, - "colab_type": "code", "id": "wR6TON_nFVM0" }, "outputs": [], @@ -736,22 +735,20 @@ }, { "cell_type": "code", - "execution_count": 58, + "execution_count": 19, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "2g1e4W2TFWcN", - "outputId": "5fc18c26-5960-4d58-a065-d9293715b39c" + "outputId": "b7bbb06f-e9db-4bb8-8191-732b97cad01d" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████| 407873900/407873900 [00:17<00:00, 23840783.16B/s]\n" + "100%|██████████| 407873900/407873900 [00:33<00:00, 12025777.98B/s]\n" ] } ], @@ -763,24 +760,336 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 20, "metadata": { - "colab": {}, - "colab_type": "code", - "id": "lxORw8EmFYcr" + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "lxORw8EmFYcr", + "outputId": "0e0b0789-32c0-4119-9b49-dab73b0878af" }, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "BertForTokenClassification(\n", + " (bert): BertModel(\n", + " (embeddings): BertEmbeddings(\n", + " (word_embeddings): Embedding(30522, 768)\n", + " (position_embeddings): Embedding(512, 768)\n", + " (token_type_embeddings): Embedding(2, 768)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (encoder): BertEncoder(\n", + " (layer): ModuleList(\n", + " (0): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (1): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (2): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (3): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (4): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (5): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (6): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (7): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (8): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (9): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (10): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (11): BertLayer(\n", + " (attention): BertAttention(\n", + " (self): BertSelfAttention(\n", + " (query): Linear(in_features=768, out_features=768, bias=True)\n", + " (key): Linear(in_features=768, out_features=768, bias=True)\n", + " (value): Linear(in_features=768, out_features=768, bias=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (output): BertSelfOutput(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (intermediate): BertIntermediate(\n", + " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", + " )\n", + " (output): BertOutput(\n", + " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", + " (LayerNorm): BertLayerNorm()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " )\n", + " )\n", + " (pooler): BertPooler(\n", + " (dense): Linear(in_features=768, out_features=768, bias=True)\n", + " (activation): Tanh()\n", + " )\n", + " )\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (classifier): Linear(in_features=768, out_features=42, bias=True)\n", + ")" + ] + }, + "execution_count": 20, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], "source": [ "#Passing model parameters into GPU\n", - "model.cuda();" + "if torch.cuda.is_available():\n", + " model.cuda()\n", + "else:\n", + " model" ] }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 21, "metadata": { - "colab": {}, - "colab_type": "code", "id": "Qy_qc5ooFaXv" }, "outputs": [], @@ -805,10 +1114,8 @@ }, { "cell_type": "code", - "execution_count": 0, + "execution_count": 22, "metadata": { - "colab": {}, - "colab_type": "code", "id": "PmDcDaHcFdDz" }, "outputs": [], @@ -822,15 +1129,13 @@ }, { "cell_type": "code", - "execution_count": 63, + "execution_count": 23, "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 374 + "base_uri": "https://localhost:8080/" }, - "colab_type": "code", "id": "3iDUGLz9FeeH", - "outputId": "b38df211-51de-419c-8099-2df1ebbecc7f" + "outputId": "4056c84c-1fe9-422d-9f7c-885536ee9762" }, "outputs": [ { @@ -845,7 +1150,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Train loss: 0.032990228633085886\n" + "Train loss: 0.4888806909322739\n" ] }, { @@ -853,17 +1158,17 @@ "output_type": "stream", "text": [ "\r", - "Epoch: 20%|██ | 1/5 [00:32<02:10, 32.73s/it]" + "Epoch: 20%|██ | 1/5 [00:20<01:21, 20.28s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Validation loss: 0.043078496253916194\n", - "Validation Accuracy: 0.9873015873015873\n", - "F1-Score: 0.8322683706070287\n", - "Train loss: 0.02619425202719867\n" + "Validation loss: 0.1323928598846708\n", + "Validation Accuracy: 0.9716203703703704\n", + "F1-Score: 0.6049495875343721\n", + "Train loss: 0.11743840376536051\n" ] }, { @@ -871,17 +1176,17 @@ "output_type": "stream", "text": [ "\r", - "Epoch: 40%|████ | 2/5 [01:05<01:38, 32.88s/it]" + "Epoch: 40%|████ | 2/5 [00:40<01:00, 20.15s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Validation loss: 0.038448549540979524\n", - "Validation Accuracy: 0.988207671957672\n", - "F1-Score: 0.8380503144654089\n", - "Train loss: 0.02107425081388404\n" + "Validation loss: 0.08609402924776077\n", + "Validation Accuracy: 0.9785383597883597\n", + "F1-Score: 0.7006039689387402\n", + "Train loss: 0.07893053752680619\n" ] }, { @@ -889,17 +1194,17 @@ "output_type": "stream", "text": [ "\r", - "Epoch: 60%|██████ | 3/5 [01:38<01:05, 32.72s/it]" + "Epoch: 60%|██████ | 3/5 [00:59<00:40, 20.07s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Validation loss: 0.04203627684286663\n", - "Validation Accuracy: 0.9881812169312169\n", - "F1-Score: 0.8351822503961965\n", - "Train loss: 0.017237978673074396\n" + "Validation loss: 0.06485400402120181\n", + "Validation Accuracy: 0.9827447089947091\n", + "F1-Score: 0.7598627787307033\n", + "Train loss: 0.0569656090810895\n" ] }, { @@ -907,33 +1212,33 @@ "output_type": "stream", "text": [ "\r", - "Epoch: 80%|████████ | 4/5 [02:11<00:32, 32.76s/it]" + "Epoch: 80%|████████ | 4/5 [01:19<00:20, 20.00s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Validation loss: 0.039959510256137164\n", - "Validation Accuracy: 0.9889947089947092\n", - "F1-Score: 0.8421874999999999\n", - "Train loss: 0.014037986324789623\n" + "Validation loss: 0.05173933506011963\n", + "Validation Accuracy: 0.9856547619047619\n", + "F1-Score: 0.8149920255183413\n", + "Train loss: 0.043735417909920214\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ - "Epoch: 100%|██████████| 5/5 [02:43<00:00, 32.74s/it]" + "Epoch: 100%|██████████| 5/5 [01:39<00:00, 19.94s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Validation loss: 0.04469061589666775\n", - "Validation Accuracy: 0.9885052910052908\n", - "F1-Score: 0.8380062305295949\n" + "Validation loss: 0.04572244201387678\n", + "Validation Accuracy: 0.9873544973544973\n", + "F1-Score: 0.8239936858721388\n" ] }, { @@ -1019,20 +1324,19 @@ }, { "cell_type": "code", - "execution_count": 65, + "execution_count": 24, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 513 }, - "colab_type": "code", "id": "ksirPywSFgdO", - "outputId": "2b4b6a14-8e14-4cde-9d0b-889e11926e9f" + "outputId": "a7c590ac-1641-4129-b296-0a03c2be35c3" }, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAA34AAAHwCAYAAAD5DL2VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOy9ebBk+V3deX53ycy31dJV1V2tlrpbrdVYYJZGSHgMBDIxksOD7DAMYCzjGLAsjyFiAoixPBOBNcSABy/gAeRhscACCxDIWAhbCwiBhKSmpW4kkFrdkrpb3eqltq711Vsy7/KbP+79/u7v3rxb3sx8mfne+UR0VNV7+TJvbq/vyfP9nqO01iCEEEIIIYQQcnhxFn0AhBBCCCGEEELmC4UfIYQQQgghhBxyKPwIIYQQQggh5JBD4UcIIYQQQgghhxwKP0IIIYQQQgg55FD4EUIIIYQQQsghh8KPEELIkUYp9T6l1PfN+rITHsO3KKWenvX1EkIIIYK36AMghBBCJkUpddP65zqAIYAo/fc/0Vq/o+11aa1fN4/LEkIIIcsEhR8hhJCVQ2u9KX9XSj0B4Ae01h8sXk4p5Wmtw4M8NkIIIWQZ4agnIYSQQ4OMTCql/rlS6jyAX1VKnVRK/Tel1CWl1NX078+3fuZPlFI/kP79HymlPqqU+rfpZb+klHpdx8u+UCn1EaXUtlLqg0qptyql/nPL+/FX0tu6ppR6SCn17db3/pZS6nPp9T6jlPrR9Oun0/t2TSl1RSn1p0op/n+eEEIIAAo/Qgghh4+zAG4BcBeANyL5f92vpv++E8AegJ+v+flvAPB5AKcB/GsAb1NKqQ6X/Q0AnwBwCsBbALyhzcErpXwAvw/gDwDcCuCHALxDKfWy9CJvQzLOugXgFQA+lH79RwA8DeAMgNsA/B8AdJvbJIQQcvih8COEEHLYiAH8S631UGu9p7W+rLX+L1rrXa31NoCfAPDNNT//pNb6l7XWEYC3A7gdiZBqfVml1J0Avh7Aj2mtR1rrjwJ4T8vjfxWATQD/T/qzHwLw3wB8T/r9AMBXKKWOaa2vaq3/3Pr67QDu0loHWus/1VpT+BFCCAFA4UcIIeTwcUlrvS//UEqtK6V+USn1pFLqBoCPADihlHIrfv68/EVrvZv+dXPCyz4PwBXrawDwVMvjfx6Ap7TWsfW1JwHckf797wH4WwCeVEp9WCn16vTr/wbAowD+QCn1uFLqzS1vjxBCyBGAwo8QQshho+hy/QiAlwH4Bq31MQDflH69anxzFpwDcItSat362gta/uyzAF5Q2M+7E8AzAKC1/qTW+vVIxkDfDeC3069va61/RGt9D4BvB/DDSqnXTHk/CCGEHBIo/AghhBx2tpDs9V1TSt0C4F/O+wa11k8CeADAW5RSvdSV+59a/vj9AHYB/O9KKV8p9S3pz/5Wel3fq5Q6rrUOANxAMtoKpdTfVkq9ON0xvI6k3iIuvwlCCCFHDQo/Qgghh51/D2ANwHMA/gzA+w/odr8XwKsBXAbwfwN4J5K+wVq01iMkQu91SI75PwD4h1rrR9KLvAHAE+nY6pvS2wGAlwD4IICbAO4D8B+01n88s3tDCCFkpVHc+yaEEELmj1LqnQAe0VrP3XEkhBBCitDxI4QQQuaAUurrlVIvUko5SqnXAng9kp08Qggh5MDxFn0AhBBCyCHlLIDfRdLj9zSAf6q1/tRiD4kQQshRhaOehBBCCCGEEHLI4agnIYQQQgghhBxyKPwIIYQQQggh5JBzaHb8Tp8+re++++5FHwYhhBBCCCGELIQHH3zwOa31mbLvHRrhd/fdd+OBBx5Y9GEQQgghhBBCyEJQSj1Z9T2OehJCCCGEEELIIYfCjxBCCCGEEEIOORR+hBBCCCGEEHLIofAjhBBCCCGEkEMOhR8hhBBCCCGEHHIo/AghhBBCCCHkkEPhRwghhBBCCCGHHAo/QgghhBBCCDnkUPgRQgghhBBCyCFnrsJPKfVapdTnlVKPKqXeXPL9vlLqnen371dK3Z1+/W6l1J5S6tPpf78wz+MkhBBCCCGEkMOMN68rVkq5AN4K4NsAPA3gk0qp92itP2dd7PsBXNVav1gp9d0AfgrAd6Xfe0xr/dXzOj5CCCGEEEIIOSrM0/F7JYBHtdaPa61HAH4LwOsLl3k9gLenf38XgNcopdQcj4kQQgghhBBCjhzzFH53AHjK+vfT6ddKL6O1DgFcB3Aq/d4LlVKfUkp9WCn1N+Z4nIQQQgghhBByqJnbqOeUnANwp9b6slLq6wC8Wyn1V7XWN+wLKaXeCOCNAHDnnXcu4DAJIYQQQgghZPmZp+P3DIAXWP9+fvq10ssopTwAxwFc1loPtdaXAUBr/SCAxwC8tHgDWutf0lrfq7W+98yZM3O4C4QQQgghhBCy+sxT+H0SwEuUUi9USvUAfDeA9xQu8x4A35f+/TsAfEhrrZVSZ9JwGCil7gHwEgCPz/FYCSGEEEIIIeTQMrdRT611qJT6QQAfAOAC+BWt9UNKqR8H8IDW+j0A3gbg15VSjwK4gkQcAsA3AfhxpVQAIAbwJq31lXkdKyGEEEIIIYQcZpTWetHHMBPuvfde/cADDyz6MMiM+L1PP4N//8Ev4o9++JvhOAx6JYQQQgghpAml1INa63vLvjfXAndCuvLYpR186bkdhPHh+GCCEEIIIYSQRULhR5aSKI4BAPEhcaQJIYQQQghZJBR+ZCkRp4+OHyGEEEIIIdND4UeWkijSuT8JIYQQQggh3aHwI0uJOH0RRz0JIYQQQgiZGgo/spSE6Y6f/EkIIYQQQgjpDoUfWUqi1PGj7iOEEEIIIWR6KPzIUhJGEu5C5UcIIYQQQsi0UPiRpUQcv4ipnoQQQgghhEwNhR9ZSkIKP0IIIYQQQmYGhR9ZSuj4EUIIIYQQMjso/MhSIrt9rHMghBBCCCFkeij8yFJiwl1Y4E4IIYQQQsjUUPiRpYQ7foQQQgghhMwOCj+ylJgdP456EkIIIYQQMjUUfmQpMTt+dPwIIYQQQgiZGgo/spQw1ZMQQgghhJDZQeFHlhLu+BFCCCGEEDI7KPxWgCee28Effu7Cog/jQBHBF1L4EUIIIYQQMjUUfivA2+97Aj/y259e9GEcKFLjEFP4EUIIIYQQMjUUfkvG+z97Hv/xTx/PfW0UxtgP4gUd0WKQcBc6foQQQgghhEwPhd+S8e5PPYN33P/l3NfCSGMUxUfK/cp2/I6W4CWEEEIIIWQeUPgtGaMoRhDlxY6IoFF0dERQluq54AMhhBBCCCHkEEDht2QMw2gsyVLGHofh0VFBsuMX0vEjhBBCCCFkaij8loxRGCOIisIv+fcwjBZxSAtBxG+sj854KyGEEEIIIfOCwm/JGIXx2F5bmM47Do9QwIuI3TCi8COEEEIIIWRaKPyWjGEYj4mdyDh+R0f4ifhlgTshhBBCCCHTQ+G3ZIzCeKzCQEY/j9Kop4jfiKOehBBCCCGETA2F35IxDOOxQBNxvUZHyPHL6hwo/AghhBBCCJkWCr8lY1ji+B3FVM9oRsLvuZtDPH11dxaHRAghhBBCyMpC4bdkjMIIWucFTxgdvR2/cEY7fj/53ofxz37jU7M4JEIIIYQQQlYWb9EHQPKIuAvjGK7jpn9PhV9wNHb84lhD9F7R/ZyUG3shru6Mpj6m933mHB45v407Tq7hpbdt4atfcGLq6ySEEEIIIeSgoPBbIrTWGKXVDWGk0U+fnaM26mkHukzr+MVaY38Ggvlfve8RfPlKNjL6xz/6LXjh6Y2pr5cQQgghhJCDgKOeS0QYa+gSp0tGPecR7hLHGvGSBajYYm9a4RfFsxF+ozDG3/2aO/Bvv/OvAQCu7U7vIhJCCCGEEHJQUPgtEbajJ6XtgDXqOQfh95PvfRjf96ufmPn1TkNO9M7E8Zv+cQvjGAPfxfOODwBgJtdJCCGEEELIQUHht0TYjl6Z6zWPHr8vX9nNjTAuA7bondaNjNPx2WmdwzDW8F2Fvp/sXR6lTkVCCCGEELL6UPgtEbaYCCyhEkTz2/ELY710/YCzdPxmJZrDSMN1FPqek17fcj1mhBBCCCGE1EHht0TkHL+oxPGbw3hhEMVLJ/xsdy7WUzp+6V2bdjQziGL4roOBT+FHCCGEEEJWDwq/JcIWYEE8vuM3imY/XriMwq8s2KYrkhA6bcBLFGt4jkLfc2dyfYQQQgghhBwkFH5LxLBix0923ubj+GkMo+USfnm3c7pjE8dwbwqhprVGKMKPjh8hhBBCCFlBKPyWCFtMBAeU6hmmjp+ecqRyloSW2IumHvWc3vGTx99zHeP4Den4EUIIIYSQFYLCb4mwA0iiknHHeSRJjtLrDqYcqZwlM+3xM6Oe3UVzZISf4o4fIYQQQghZSSj8lojcjl9ZuMucHL/kupfHwSq7710R43Qah07cV89R6LnO1NdHCCGEEELIQUPht0RU9fhJ0Mu8Uj2Lt71oognqHHZHIX75I49XCkQz6jmFsDWOn+NAqaTSgY4fIYQQQghZJSj8lghbTIgTF8casuY2mkMIi7hr87juruR2/BqE3x9+7gJ+4r0P46Fnr5d+P57BqKc8Rr6rAAAD32WqJyGEEEIIWSko/JYI23UTp8t2vOYxjrnsjl+T8Hv22j4A4OZ+WH5dkuo5mibcJXlsXCd5u9DxI4QQQgghqwaF3xKRc/xSsWG7X/MY9TQdgUskZMIJhN+563sAgO1hufCbxainhOt4qePX9yn8CCGEEELIakHht0SMLHEiYiPv+M1h1DOUcJfZXPcwjPCuB5+eqh5ikh0/cfx2KoTfLFI9TZ2Dk456ehz1JIQQQgghqwWF3xIxKunuC6M5j3qmjuKsdvw+9uhz+NHf+Qt87tyNztdhi724peN3s9LxS/6cqsdPUj3TRE86foQQQgghZNWg8Fsi7FHObMcv+9o8xjFNuMusHL/0PkwjtKJ4/HGo4vz1dMevSviljt809QtyDH7q+PU9d6nqLybl7//yn+Hdn3pm0YdBCCGEEEIOEAq/JSLn+KV/zzt+sxV+cazNWGWT8NsZhni4hYsXzqBzUMRo33OMcCtjP4hweWcEoCbcJT2evakcv+Q6XBn19J257FseBLujEB9/7DL+4ulriz4UQgghhBBygFD4LRHDklRPES5rvjtz4RdM4Cb+5ie+jL/z1o8ZQVqFCLVpHES5z33PyQnfIuL2AdU7fjOpc0gfJ19GPT13qrCYRXLxxhDAdI4sIYQQQghZPSj8lohcnUMqeKRuYaPvTTWuWIYtqpp2/K7ujjAMY+PGNV1n0+Vqr0OEn+/Wpno+m+73AdWpnvLz042eFlI9vdV1/C7dTITfNPUWhBBCCCFk9aDwWyKGYYx0mtDsuYno2OjPwfGL2jt+InRsl7CMtqOj9deR/Gzfc0wqZxni+A18pzrV09Q5TDN6Kj1+WYH7qoa7iOO3S+FHCCGEEHKkoPBbIoZhhPWeByBzzMT9Wu95CK2dvFlgu3KNwi8c3zksQ4TaKJp+p67nObXhLudS4XfP6c3KcBfRjdOleqbhLm5W4L6qo5IXt5PHbJqdR0IIIYQQsnpQ+C0RozDGes8FkDlVIjo20q/PMtnTdvyGDaOekmLZtOMnQi0Ip+/x63tubZ3Ds9f2cGLdx5mtPm4Oy4VM1uM3g1FPxxr1XFXHb5s7foQQQgghRxEKvyViGMbY6KeOXzruKHUO8vVZ1giEHRy/pl1AEWpNQrL2uOL2jt/tx9ew2fdwcz8ovcwsdvxEIHtO8nZJRj1XUzjJqCcdP0IIIYSQowWF3xKRc/wKo54b/eTrs3SaRh12/JpGPcMZ7PiJq9j3nFynX5Fz1/fxvOMDbPY97FQ4frNI9QxLwl32gxi6Zv9wWWG4CyGEEELI0YTCb4mwhV8wNuqZOn4zTJOcpBzejHo2hrukITAzcPwS4Vfn+O3h7PEBNvpe5Y7fLBw/U+Auws9Px26nuI+L4uKNZMdvGiFMCCGEEEJWDwq/JWIYRuh7LlxHGQElQksE4SxHDO09vKYwFnEam2oaRAvNpsevus5hbxTh2m6A551Yw+bAw84oLN0HlC9N07sXmlTPLNwFmK37elBc2uaoJyGEEELIUYTCb4kYRTH6ngPPUcbpM6meZsdvhuEuEzl+7Zw8EaxTjXqaHr/qOgfp8Lv9+ACbfRdaA7sFMWMLwZmMejp5x2/VAlKCKMblnREAjnoSQgghhBw1KPyWiFEYoyfCrzDqudlR+IVRjPd/9nzpPloQTj7q2dbxm2bU0zh+rmN2HYtIh18S7uIDwFiXny0a96cQOmV1DsBsx24PgufS/b4zW33sBVFtYiohhBBCCDlcUPgtEUMRfq5jxgujKUc9P/roc3jTf34QD5/bHvuenZjZJCizcJd2jt80zmS2U1ed6vnstczxk+Cb7f2C8LMdv2lGPePxAndg9UY9JdHzrlvWAaze8RNCCCGEkO5Q+C0Ro9Aa9Yzzo54bHR0/GenbHY2Hn0yU6ikF7g0ukUn1nMrxi+E5Cp6rTCpnESlvP3t8gK1B8tgUHT/52Y2eiyDStUExdWSOX5bqCazeqKfs992ZCj/u+RFCCCGEHB0o/JaIzPGzdvymTPUUAVY2omlXM7QtcG/c8dNS4D6d4+c6Cq4lgIucu76HUxs9DHzXPDbFZE/5UdmP7CrUio7fqoa7SHn7naco/AghhBBCjhoUfkvEKIzRc114jjPm+K33u1UIiOArE2zyNUdNEu7SsOMXTe/4hZGGlwq/qh2/c9f3cfuJAQBgc1Au/MThkzHZrsIvKOz4ZaOeqyWcLm7vQyngBSdT4ceAF0IIIYSQIwOF3xIxCmP0/dTxi/M7dZnjN9nJemgcv2rht9H3Jihwb+f4TVvn4LnJyGtVquf56/s4e2wNQBZ8c7Ow4xcb4Zc6fh2PKSqmeq6w43fLes+Mxq7aqCohhBBCCOkOhd+SEMcaoyhGz3VyI47Zjl+3QJGgVvhlY6R1Qk1rnY16NuzJiUiarsA92fFzakY9bw5DHEsFjAi/nVF5qudG6vh1dbiyHj8RfulzsWLC6eKNIc5s9bEmj8eKHT8hhBBCCOkOhd+SIKORfd+B72SpnmOO38TCT+f+tDHX3XdrRzPDWJt9uSbHT4TaNG5YlO74eY6qrBwIIw0vDVuR4JtiqmdcCMbpPOoZJ6OnSkmq52o6fpe293HrsYEZfeWoJyGEEELI0YHCb0kQESGOX1Sx4zfpXlm949du1NMWOE1OnoitqQrcZcdPJY5fWQdhEgCT9er5ripJ9Uz+3Oj42AnJ6Kky/5YC91Xr8bu0PcSZzb7ZUdyl8COEEEIIOTJQ+C0JIpRExIhDFxX21CYVG2HN6GXbUU97pLEp3KXu9toSxRquq4ywKzP9wjg29QpKKWz0vfFwl1Qwrvni+HU7piCK4TnZW8XUOaxQuIvWGpduDnHrsT7W/OnCbgghhBBCyOpB4bckmFFPzy11/HxXoec6E6dliqAblQi2oOWop+34NRe4zyDVM9bwHMe4bGX9e4krmL18N0uEX1zYj+xc5xDlHb/BCjp+V3cDBJHGrdzxI4QQQgg5klD4LQniqiU9fo4RZaY83HHQ95yJxYYZ9Sxx9Oxy+Lajnk0F7tEMRj1lx89R1cIviDLHD0iF335VnUPi+HUVOiJEhSzVc3WE08XtpPD+1q2Bcfy440cIIYQQcnSYq/BTSr1WKfV5pdSjSqk3l3y/r5R6Z/r9+5VSdxe+f6dS6qZS6kfneZzLQOb4pTUGxvGLoRTgOAp935lYbITW9YzdZirO1ntubVCJfZtNTl6W6lkvEOtIRiuVqU8oq3QQcShs9r3KVM9N4/h1E6NhejyC5yg4qvv1LYKLN5Ly9luPZTt+dPwIIYQQQo4OcxN+SikXwFsBvA7AVwD4HqXUVxQu9v0ArmqtXwzgZwD8VOH7Pw3gffM6xmVCnDzj+Fmjnr4JMakXaGWMaorXpTah77kY1QhK22UMmwrcZ+T4eW5S5wBgrMRda524cG728t0ocfwkFMb0+E3j+FnuolIKA99dKcfv0nYi/M5s9tH3HCjFHT9CCCGEkKPEPB2/VwJ4VGv9uNZ6BOC3ALy+cJnXA3h7+vd3AXiNSjPzlVJ/B8CXADw0x2NcGsRJ6xnHL6tzEGer5zkTCz9x+sqEWBBp+K6Dnle/O2gLhIOoc5DETnHZim6l2Xu0Hb9BSbhL+mNSXzCN8PPd/Ful3+G5WCQXtzPHTymFdd/lqCchhBBCyBFinsLvDgBPWf9+Ov1a6WW01iGA6wBOKaU2AfxzAP/XHI9vqchSPV14jjLOmu029T2n1pkrIwjrUj1jeBIa07bOoXHHr7o+oi1R2pvnVox6iqvo2jt+vTLhJ1UY3ToQBVt8C33PXSnH7OYwgOco436u9VyOehJCCCGEHCGWNdzlLQB+Rmt9s+5CSqk3KqUeUEo9cOnSpYM5sjkhY4PJqKfKdvPSTjugm8sUiHNYEZDSSx2/WFe7eROleqY3M1WPX5wILSP8CscuotK3Uz0HHnaGeSETp4Jx4DnpTt404S554TfwV8vxK0smpeNHCCGEEHJ08OZ43c8AeIH17+enXyu7zNNKKQ/AcQCXAXwDgO9QSv1rACcAxEqpfa31z9s/rLX+JQC/BAD33ntv9zSRJWBkFbh7jmMEVr6o3O2Q6lm9cydioJemVI6iOLc3J9i7bE2hLeL4TVPnEKWjlSL8inuF8m9byEiPXxzrbDdQnEFHTSV0wiguGfWc/LlYJMVk0jWfjh8hhBBCyFFinsLvkwBeopR6IRKB990A/n7hMu8B8H0A7gPwHQA+pJNEjr8hF1BKvQXAzaLoO2yIe9T3k922rMA9qy3o+w52CuOMTYiALBu9HKWCppeKmlEYY71XcmyWwGka4RSxFcV6LHmz9THHGgNfwU3rHOLCqKe4mLZI3UrHOXeDCJvp3+XnnFT4dS1cD0vuR5eE1UUSpmO9Akc9CSGEEEKOFnMb9Ux39n4QwAcAPAzgt7XWDymlflwp9e3pxd6GZKfvUQA/DGCs8uGoMLQdP9eqc4gy0dFzO4x61gi/0Ap3AarHM+U2lWqf6ll1m22Q8VYRKsUxVbkNe/xyIxV7drKnCD9XKQw8Z4o6B53rDASAgeeuVJ1DUHD8lmXU8+FzN/Caf/cnuL4XLPpQpuapK7t46x8/atJkCSGEEEKWiXk6ftBavxfAewtf+zHr7/sAvrPhOt4yl4NbMkaW4+c6jkmytBMl+34WwhJGMa7uBjiz1a+/3qi6V0/68nqmkLxK+CUCYbPnGbetClukDcPYdMZNgoy3iuCN44pRz0KqJ5CEmAADAFmqp5PWL3Tf8Ytzogno5r4ukijK7ymu+S6u7Y4WeEQJj5y/gccu7eD89X0cX/MXfThT8b7PnsO/+cDn8Q9eddfK3xdCCCGEHD6WNdzlyGFGPV0Xvh3uEmeJknaP3zvu/zK+9d/9SXO9Qo3jJ3UO/Ubhl3x9o+817vjFM3D8ZLxVRj2Ljp8Jd7FGPaWk/aYV8CLOoOMgFX7djicoBKMAq1fnEMT5Uc/1JRn1lDHiYmXHKiL3pRhGRAghhBCyDFD4LQkm3MVLnC5T5zCW6pmcrD987ga298NG8VE36hlEicDqN416BiL83NY9fnXX14Ts1FWletqhLcJmP3FYKkc9faez4xeVpHr2p3AQF0HxPixLuIu8fg+DWBpFk4vYT3zpCj7yhdVOJCaEEELIajDXUU/SHrvOwXfzo552j5+cKD91dRdAIq42aqY9g5pRzzCO8zt+lXUOUSoQ3RapntMLPxEpcr/H6xzSAvdcqqc4fiXCz5lu1DOIYpOsKqya45ckuFo7fj0Xe6PFH7+87svqRlYNeT1MYl7+3Ie+iBt7Ab7ppWfmdFSEEEIIIQl0/JaEURib0vKxAnepc/CzCoGnr+4lP9fgwNU6fmFa5+C65hjKGIYx+p6MoLZL9ay6zTYkgTYOnIpRTzkGe+9uSxw/S/hlo57Tp3oWw13ssdtVQPY5hTXfxd5o8TuKh2k80uzfTqD8hmG8Uq8jQgghhKwuFH5LwiiMzcil5yQ7flprRHF2wp6kekaIYo1nr+2Zn6sjNI5fifArOn414S59z4HnOq3qHJrCYpowjl8q7MbqHEp7/BLxulPi+Dlm1HOK4xnr8es+OroIoji/pyijnotOoDTjkQ1O8iog7uUkInYUxp0/ICGEEEIImQQKvyVhGMZGMInIiGKNwKpz6HsOYg08e20vK2Zv7fiVp3rmRz3LhcwwSESp3S9YRaQ11tIkz64l7mGs4boKYugVRUFW55C9fE2dQ87xS/50p0z1LLplQBIWs0pOTbHOYa3nItbdn6NZYcYjD0EFQhf3MhF+q3/fCSGEELL8UPgtCSNL+InQC9MSdLvOAQAevXgz93N1SP1CdY+fyhW4lzEMY/R9Fz3PaZEiqrHeS4Rf0DncJc45fsUTaTkG28Hqew58V1WMek6X6hlGJeEuXlKtsWjHrC22cwzA1GzsL3jPbxgcoh2/qIPwi+j4EUIIIeRgoPBbEpJxyuRk3LeKy8MoX+cAAI9dyoRf00ljEKbOYIkIG0UxPMvxq+vxE8ev6QQ9noHjF0WFVM/iqGc8Hu6ilMJG3ytP9XRUWrjefcdvbNTTn26c9aApVlKIOF90smeW6rkaj2MdWTUFRz0JIYQQsnxQ+C0Jo8h2/FKnK9K5YBHZAZzI8asJdwkjjZ7V41cf7iI7fvUntWGsjZs0TZ2D7zpWnUP+eqKScBcA2Ox7pTt+09Y5hHHJqGcqwocdXcSDJiqOevrLJfyO8o5f1/cJIYQQQsgkUPgtCaMwNiOXIvSCOE4TLtNwl1Sg2Y5f2x2/MhdCdtca6xyCLNWzTbiLGfXsXOCuTbopMC4KRHy6BTG22fewXTLqqdIdP3FQJz6esgJ34/itRsBLGOUL3EWc740We/yjcPLxyGWly31JRj1X/74TQgghZPmh8FsSkj26/I5fFOvEbZIdv9RlevzSjhGHzY5fmupZcrkg0vA9p8WOX4S+n/YLthB+a6nw6zoGKQ6b1DkUgz9C0+OXf/mu91zsjspHPcXh2u9wTJJ+ataXdzwAACAASURBVCPPRde9wYMmiMbDXYBlcPxSl2xFdiXrMO7lpI4fRz0JIYQQcgBQ+C0JQ9vxS0/QgyhOe/zyo56Xd0a4+9QGgBZ1DulY5Kgq1dN2/JpGPZ3mUc9oylHPONaIdSLWPLehx6/gwnmuk3MH86meyX3sMu5pu67CYMUcvyjWYz1+wOIdv+FRd/zCGFEa4kQIIYQQMk8o/JaEUZqcCRQcv8gucM+ernvOJMKvTohprTPHr3THr22P32QF7iIquoywifPj2eEuY6me2WVsPEflLhtbqZ79jkJH63TPcizVczpX86AJ4nisxw9YAscvOLo7flpr4/Yx4IUQQggh84bCb0mwHT85QQ8inXNqRGwAwD1nNgFUd+8BeaesbEQzSXpM0jqVqtnxMwXuLXr8rB2/UQc3LDT7ew5cVSH8jOOXf/m6jsqNDOZSPX0RapMdk+kMLClw73J9i2LM8eslx79w4dchEGVZmdS9tN9vFH6EEEIImTcUfnNEa41HL263uuwoFVcAcv11YRzDdfPhLgBwz+nmUU85mXTUuPumtUYQx+i5CkolXX6Vjl+Q7B8mo57No6WDKRw/I+osx6846inXW3Thio6fiEBHWTt+E+7kyW2Pj3qu1o5fGOUrKdZ6SeH9/pKMeh6GHr+RuS/tXhP2+40BL4QQQgiZNxR+c+TtH38Cf/OnP4Jnru01Xlb26ADb8YtzY4Z9S/i96FZx/KpPGKXDb6PnYRTly8ajWEPrzMnqeU7l2OJ+kHQMJgXu9aOlsc6CQ7qEVmQOW7bjF4+NelY5fvnjM6OeU+z4iSDxi6meK+b4SYKrsCyjntle3GoI6Drk/VMMI6oiL/xW//6T5WYUxrh4Y3/Rh0EIIWSBUPjNkW9+2a0AgA989nzjZUdh1uPnFXb8pNdPxIajgLtuWTc/V0WQnkyLEMuNfhpBk1139ahnbBW4V9+eiLY1v/v+mxyX5ygz6jke7lLuwo05fvH4qOfEjp+IzEJnoKlzWBHHL4r1cu74HaJwF/kQoO2+ov1+Y5cfmTfvfOApvOanP3wo3muEEEK6QeE3R154egMvP7uF97cRflaBuzhZYRwjjOOswD09Wb/9+BrW01G9NqOeZb16ctIp11016qm1TqsmXFPgriscjcjaqWvT+Vd6HUasZQXuY3UOFS6cWxCmcv7tKmUK1yd1/GQEr5ggaq5vpRy/7O0uHyIsPtUzFUsrfjKqtZ441XMSx+/3Pv0M3vC2+7sfIDnyXNoeYns/pLtMCCFHGAq/OfPaV5zFJ5+8govb9SM2SUl63vELI52rEpDvP//kmhGJdf8TF+dB9rnsPaJiF17PKxd+IhD7nmNGTqtO0iPLravbGawj5/hVFLhXuXBuwfETgeo4Wf3CpA5Xdp8OgeNnOaSOk4y/dqm3mCXy+K26CxGmNSRA+07CSXb8/vLp6/j4Y5c7Hx8hMk696u81Qggh3aHwmzOve8Xt0Br4g4cumK9dKNmzyDl+lsAKY20VuIvwW4ebCqM6cSWibaPE8QvMnlwWHFN2XTKK1/cc+OntV42y2SOYVdfXRBRl11FZ52CJQxuvkOopP5fs+HUbPy0+TsLq1TnosZ3INd/F7qIdv+hwhLvYr/W2J9b2a6exjzOq7/v7lY9+CfdRGJIa5Pd22w8mCCGEHD4o/ObMS2/bxD2nN8y458/+0RfxDT/5R3j04k1zGTmpEzEhIkPG4ETgDPwkYEU6/JrGKTPHr1r45Ry/kusSR6bvu+Y4goo9v9gSfr7bnABaesxWObtJNy2OekYajkpcKxvXUUY42j8nQhSYfJeqSmROUwi/CIqOH5AIv0Xu+HUZj1xWbBHXacev4b0SxNV9nADwM3/4BfzmJ77c6na7sjsK8ZdPX5vrbZD5Ia5yMSyLEELI0YHCb84opfA/vuIs7nv8Mv7Tx76En/7DLwBALl1NTvp6hToHCSIRIei7Dv7Lm74R/+gb704u71YncQLZSeKGjHqG2vpefk8uua5xESBf63uOEYlNjp83heNnu4YyXVkUBUkZ+fhL13NVzjmyUz2lI3HSbsGoojNQrm8VHD+t9Vi4C5B8ILBI4ZcTSyt+Mmq/d7qNeta/juQDjbLX2yiMsT0McX7OiY2/88DT+Hv/38dX5sMOkkd+l636e40QQkh3KPwOgNe94iyiWOMtv/853HFiDQCwY43YyQmgiAkZcSw6fgDwlc8/jo1+IuR6nlvrFIyFu8S2K5F3/Pqe2zjqaddMlGGEVir8hl0cvygTj3afYfEyxQ4/YHzHT24+5/hNeExBVO74SfH9KtQ5BIV9TmGt5y60x88WMavuQnQZ9ZxE+Ml7t+xyV3dHAMpHyGfJ9b0AQaRX4sMOMo64xqv+XiOEENIdCr8D4CvvOI57Tm/gntMb+Nnv+RoAydiUYMRVOj4oJ+ji+LlO+dPUcxWCWsevetRzVAhIqdzxk1FPzzXHVXWSmnP8XKf22Kqww1REaxU/oY5iPVblACTpnTnHzxS4o/uoZ4XwA5LR21UocLdrLWwWPeppi+ZVdyG6uJeTCD95HZa9fq/sJMLv3PX9ysTdWSC3TeGwmsiHfdzxI4SQo4u36AM4Ciil8NtvejXWey5u7CWCb2fY7PjJSFWxtkCo2ssTzKhnf3zUU04ke14W7lL2Sb4Z9fQd+EF5yqZgh6k0HVsVsuPnugpKqdTFy19PEMVj7hWQCOS4IPyUSh5/zwGU6rLjl3dGbfpe+XjssiFu0fieooubw7DsRw6EvEu2/AK6DjvdNWr5us/3+NWfjMvrsOz1ezUVfqMwxrXdACc3eq1uf1LkeFddpB9V5Hlru4NKCCHk8EHH74A4vdnHes/Dej9x33aGtuMn4ir5ngi9zPErF35+Q2WCCD8p6x6VhLvkHL+ycBd71NPJ+gXLMG6d273Owa6EAGR8M3+ZMBrfV5PbDXOjntqUwKt0z2/S8dOqsnggFX6r4PhVuJZrvrvQHr/DtOM3iuwdv5Y/M8mopzh+JZe7nAo/AHPd8zOOHx2jlUQEH58/Qgg5ulD4HTAStLJTMupZ6fhVjXp69cmZcrJYVuBe3PvqVwi1TPi5RpBWuROZSHKmSPXMCy1XjTt+YazHevXkZ3I7flrnkj+7BM6YUc8SoTnw3ZXYdwoqAmrWeu5CgzpyLtmKC7/8fWnp+E006lnj+O0ejPCT1/qqi/SjSsgeP0IIOfJQ+B0wblqcbfenycn3oLjjlzqBVY5f1XimICeLIvzsEZ+szqGhxy/IUj2bHD/5JNlVU/T4FQrTPUeNnWiGcVzu+Dkqd2yx5fjJfZjlqGfPm28B+qz2tar2FBfd43eodvws4db2vgxLHPgq5DrLHL8rluN34fr8Hb+Io4IrienxW/H3GiGEkO5Q+C2AjZ6XG/Xc3k/+vjXwAVipnoU6hyJNo54jI/zSHb/cyWle0DSNeg78rMA9qKpzsMrXm0RpFXKM8hg4jhoLkwij8U46INktjHUWPhHrvGjuMn5q36cifd/F/pwcv3/4K5/AT7734ZlcVzaCO+74LUudw6oHhtiOX9v7Yr8Wm16XQY3jd2VnhI2eC6XmPOrJcJCVRj484PNHCCFHFwq/BbDezzstErCxNUgEmm96/KTOofxp6jeMeoaFUc+yMAnP6vFrHPV0JNylfsfPlVTPDqOexR2/MsevKtxFfkZObKI4CXcRugTOyG2XjdsOPMc4orPmixe28fb7nsTlm8Opr6vo7gpr/mJHPUeHaMevi3uZE34NLpq8l8veU1d2Rrj12ACnNvpzrXSQDsxVD+I5qshrh44fIYQcXSj8FsBGz8ulKd5MHb/NNH3TdfM7flWOX89tl+pZVucgjl/PbahzsArcPbfe8RPBZQrcZ7Hj56ixMILKOof0cYqM45e/XLcdv2rXdTBH4TQKY4zCGL/5iS9PfV11dQ5BpPFr9z2BV/7EB/Fzf/TFqW9rEmzHb9VPRuff41dd53B1d4ST6z7OHu/j/EGMelL3rSQc9SSEEELhtwA2+l6ux09E4Gbq+Hkm3KU8hl9om+opo55lO36eJfzCWI+Nqdk9fqbAvTLVMxvTnDbV07eCborx40Gsx8YWAcvxi7MTHHvHr4vwCwoOpM3aHHv8RDT/2n1PdnocbbIS+vFRTwD4sd97CBe3h3j8uZ2pbmdSDtWOXxfhZyWBNnVe1oW7XL45wi0bfZw9NsD5G9M7xFVkdQ5UfquI/XuREELI0YTCbwGs99xcj5/s+EnipxF+LcJdqtw3YDzVM1/nIAIrC3cpXgbIl8uLO1jd4wdzvL6nao+tijLHr3iiEkaxGTu1cU34TOb4OcUdvwntiqgiERNI9h7359TjF0QxXn52Cxe3h3jvZ85NdV1hRY/fK194C159zyn8wj/4Orzw9MaBnxDKhwo9z1mZ8cFPPnGltAKjk/ALY1O10rrAveRyV3dHuGXDx23HBjh/fa/VbXchc/woHFaRgKmehBBy5KHwWwAbvXHHb6Pn5sQOYBe4V9c5tHH8TIF7SYqgCXdJ/ywGsogr03Md4/hV7fiFOcfP7ej45UWK56ixMIIwrujxKzh+cYwxx2/SwJmgIhETKB/1/PkPfRH/4nc/M9FtVN3ut778VrzozAZ+5WNfmirhU4Rw8TH7quefwG++8VV47SvOlgrseSPPxUbPXYmT0et7Ab7rF+/Duz/9zNj35LVetpNaxSiM0fOcdCy6/meCigJ3rTWu7gTG8bu6G8x1/BigcFhVOOpJCCGEwm8BrPfzjt/N/dCMeQJJ2bjnqHYF7nXhLnGhx88O0yj2+InjNyb8YvRcB46jzKhgUHHikAt36VjnUEzRdMrqHKK4NPBG3D3TV6U17IeuaTS27niqdvyK7s8DT17FA09cmeg2ikSxRhRr9D0Xb3jVXfjLp6/jS1OMYRaf6zKKVRgHgYSFrPe8lTgZ3RtFiHW2k2sjH5CsTSBiR1Eq/FoEIVU5fjeHIUZRnDh+xwcAgItzGvccUvitNEz1JIQQQuG3ADb7Xq7A/eYwNMEugucqy/ErF35NvXTyvSzcpWzHr2HUM4iNKJTjqNpHyqd6KoyieGKnKiq4U15ZnUNrxy8/6tmlxy9zIMtGPcfrHPZGUac0UxvjxnoKd55aBwDcKBEbbbGd2CocpQ48tEOExHrPXYkdP3leynZcR2EM31XwXae1MJIPVXxXtS5wL773ru4EAICT6z2cPZYIv3lVOpg6hxV4rsg4IZ8/QgiZGR946Dz+2Tv+fOochoOGwm8BrPc87No7fsMQm2mHn+A52ViiW1Hn0OSqhXFsglaA/AmriEA71RMoc/wi9NNiedlzq3KG7CqGXkPnXxVBXHD8VFmdgy4VYm5B+EVlqZ4TqpviLqTNwE8ef1uY7gdRp93G/G1miasDzzXX25Ww5j4InqsOfM/OFn6rcDI6MuJr/FhFxLkTjnr2PQd+C8fPpHoWLndlNylvP7XZw9njcxZ+Uzh+2/sB3vC2+/HUld1ZHxZpiXH8VuC9Rgghy86jF2/iv3/mXK42bBWg8FsAGz0Xoyg2J1I7wxBbNY5fdapnvVMQRNq4EED+hNU4fo70+KUBMCWjnv1UfBjHrzLcJfm6o1Slg9hEZI5LxOb47lkUx6WPSWOqZ5dRzxq3bJAGc9h7g3tB1KnGwiawRjP7/vTCL3Niq9/ukwiWWTEMI/MhQVVg0DIh75myDz6SD0jcZCe1pYCWHT/fbd49rUr1vLKTjHWeXO/httTxuzCnSgcj/DqMCj7x3C7+9IvP4S+fvj7rwyItkddtsR6HEELI5JjVpBVTfhR+C2A9FXmyH3Zzv2TU01HmZLC6x88trWAQgiiG7yQuhOvkRWIYxVAqEzRVjt9+EGWjnpKa2VDg7uXE5qRCq5jqOT46F0blo56u2fFLLq818qmeXXr8CvUSNoP0cdmzRNleMMNRT9fBIHVbp6mNKIr8Mlw13pc4b2SMeBHBMl2Q107ZBx/i3k0yMmt2/BrSeQF7xy9/uSvpqOctGz0cG3hY8925j3p2+YBAqiuGc0rBJc3Ia2gVPmQhhJBlJ4pjOCp/nrkKUPgtgI105+5muud3c5gPdwESxytz/MqfJt9LXmxVDlMQxfBTceIVhN8o0vAdB0oVd/zyJ2bD1JUAMgFadZJqRNs0jp8RWqnwU+OjSUEclwoxeZxsx89+P3YZ9SyGzdjI7qTtxu2N4onFbhERGD3PsVzF7Dau7wb4J7/+AC7fbBfiUZXqaVPWlzhvRPh4jrMSgRPG8St5Dcn7ZJKR2ZG949fwmqlK9by6k4x63rLRg1IKZ48P5j/q2eF1MgqrC+jJwSC/t+n4EULI9IRx+drRsrN6R3wIkHqF3bS4fXs/GHP8XEcZJ6nKqZH9vErhF2prlDPvKoRRnNv5susc9kYRnr22Z/4t4sMv2RW0kRMKe6+wq8MmQstznLHRuijSpY+JaEETW641nNyo5+QVE2GNWzYoGcOcxY7fyDh+qvQ2Hnr2Oj7w0AU89OyNVtdnhF/DqOdiHD93IWOmXRDxUubo2u7lpHUObXb8jONXeP1e3hnBd5X5/XHbsf7Eo54Xbuzj8+e3Wx0v0G3UU+7ftGPQpDsm7ZhPASGETE0U69rQvGWFwm8BbPSTk/mdUQStdWmqp+8qyPlV1QurqoJBsJ0x38ufXAZRnCslF4duGMT4x7/2AP72z30UQRRjaI96NhS4m+oDx5na8ROR4jhJH1/+flWNeiY/IwImjkvCXToIUc9Rxhm1kd1HEehaa7PjN03vXj7cZXzUU26vbf1CaAnJKha149f3ZdRz+c9Gs1TPklHPKBWxE4zMiuPZVMuitTbPTVEgXt0Z4eR6z7w+zx4b4NyEwu9n/vAL+KfveLD2MnGspwoHMcKPjt/CEKf2oGtbCCHkMCLnh6sGhd8CWO9ljt9ekHSDFUc9bcFS1b/mN7hqQaSNACsGwQSxzl2viLtf/Mhj+Oijz+HKzgh//uTVJNwldZ1cR0Gpmh2/9ITXcTC14yd3v8zxq+rx8wo7flWpnpOIsrDmE53i/l0QaXNSPI2IkhCeZMevZJxUhF9LZ7HoopbhLajA3bhkK7B3VD/qGaE34X2RUc+eV+/42a+lsXCX3RFu2eiZf992fICL2/uVe79lbA9D3Nirrwuxhek0wq8pxIbMj4DhLoQQMjOiQmXYqkDhtwA2UuF3cxiaMuhxxy97aqpO2LPKhPKTqUQgKXN99slbEBZGPdPr+rPHr+CbXnoGnqPwJ1+4ZE7OzXE5zljAhGC7dVVhMU1IYqc4GK6jULy5pnCXyJzgIDfq2e/gQoaRrhTea7J/lwoxO+RlmoAXM+rp2cIvuz75e1tx2abAfREBK6M0MXYRorML8louE3YS7lKWQlt5fbkC9+qfsW9vrM5hJy/8zh4bIIi0qXloQxDGjaErtmDr8lzJ7wwKv8VhRuD5FBBCyNSEFQnzyw6F3wKQUc/dUYTtdM9vq9bxqxd+1Y6fNerp5iPzwwrH78xWHz/zP/81fN1dJ/Enn7+UjONZws9zVaXjZ9w6x9oH7BCmYt/3sjHA4rGbYxPHT0IMiuEuHVzIMI4rQ1GMKEtPmm1XrqzrrS2BNZrpOgq+q8xtAJnAbPvYtilwX4Twk0CUlUn1rNlTy+5L+6CabMevvpbFdryLt311Z4STlvC7dSupdHiuZfBPcv0aw4bUWPs90ynVM+So5yKxx4VXYayaEEKWHe74kdZIuMvOqNrx81o4fn5TuEva45dctpjqmRc0tx0b4BtfdAr/73d9NU5t9vEtL7sVD5+7gfPX980uG5CIq6oTv7jC8dveD/C//danWqVQFmemy0bnpJi+yFiBe8mOnxxTW4KKIBkgE357ozj9MxNn04RY2Dt+ADDw3HyATHo7bcWScfxqwl2SkdoF7Ph5TuL4rcD4mbhyZY6fCXcpSaGtwi5wr3tN5hy/klHPU5bwO7nuAwCupjUPbQiiGKMorh0PtR3BScZI7dsAKPwWhf2apONHCCHTE3HHj7RlPa0B2B1GuDmsEH4tdvwmdfyKPX4963oHvovf+Mevwje++DQA4FtediY5xlESwGHfZuVoqbVLZsJiohif+NIVvPvTz+LBJ6+W/pxNUax5hbRJrXUiaGuEn73jl0v17DDqmYyelj/+2Y7fbEc97R4/AOj7bmHUc9Idv9Txqwl3cRzV6YR+GrIdv9UvcJdwF2+C+zJK34N+Q82InaKbc96iGNd2A5xcz4TfifTv1yYZ9Wyxfzet45elerLHbxGEOeFH5UcIIdMSxrr2vGpZofBbABLusjMKsS2O31iPX971KqPfGO6SuXrJjp+2vle+Jye8/OwWzh5LxsZyo541J7axLfysAvcnLu8CyAujKsJCR59TcBizkvjxl65x/KpSPbuMetY8TmuFUc+Z7fhZ4S5AIjCHZeEubR0/48TWh7scuONn6hy67Y0dNHUF7sMgmnhsVUY9ew11DmHufZtd7tpe4uqd2rQcv43U8dudxPGT/bvq92cu3KWDOyuPXdNIKZkPZb9DCSGEdCdijx9pi+soDHwHO8PQOH5bfT93GVtsVJ2w+ybcpfx/5IEVTFIsibbdwDKUUvjmlyauX27Us2YfyRYYtrv25cs7AICdYbPwK3X8rBOVuoRKU+BuFRVPPepZY+X3C8Er+6PZhrvI8Q58t3THr32dw5KGu0RxWudw8GOmXTB1DhU7fpJQ2mnHr2YntGrUU8rbbcdP/n51AsdPdnb3a0SZfbtRh9e2/I5ij99iCHPCfYEHQgghh4S61PdlhsJvQWz2PeyMItzcTz6ZH3f8kqfGreiQA+wC93JBNRbuEheEX8MnFTLumUv1dJ3SHjMgGyFyVL7AXRy/3VF9ZDyQOmy226nyTpT8vSzwZnzUE7nHzoyfTiD8ojgudReB+lHP0TThLmFhx893phv1jOR5qb6MqxYQ7hJE6Lnpjt8KjJ+JaKlO9WxfRi+9eL7bXOAeVIS7XEmFn53qOfBdDHzHiMI2jNo4fuF0woE7fovF/nBwFd5rhBCy7EQ1GRDLDIXfgljvedgdhthJXSJJ+hTkxVT3aUKTgxXmwl3GRz19r/4F+9dfchrHBh5uPz4wX/NrUj3ly56jLDcyxpOp42eHn1QRFWami06U3HZpj5+bD3eJYw1bHzbVX5RRF+7Scx0oNccdv/T5GQt3MXUObVM9k9dB1QcIQLL/d/DhLrEpcJ/ktu977DJ+/b4n5nZcVYgrF5Q87pLq2VbE2q5uU4F7leNXJvyAxPWbZNRzYsevg3BgqudiYbgLIYTMllV1/Lzmi5B5sN5zsTOKsL0fouc5uXFKIBMxZSEmQpbqWTXqWQh3KQRDeP36p//YwMfH3vytpncQSARX1WhpFMdQKtnLE7dqdxTh6at7yd9b7PgFhZnpYi9aneMnQS4iiIpjo007kWWEUXWdg1IKa34myvZmNOpZDHcZ+G7OLZXbmWTHr+mXU+L4HewZ4TB1yRQmS4p814NP40+/eAlvePXdczu2MqpGPbXWabiLA8dRrU6sh5ar228ocJfv9QshMNLVVxR+J9Z7HcNdqt+fQzsYigXuK4f9+mKBOyGETE/EHj8yCRt9L93xC7BVIsDsUc8q+k2pnnGc3/HL1TlUF5PbbA18OIVOwaqT1EhruKn4EuH3xHM75kRxd9g86hkVqhocVXT8ZMevusfPOH4zSPUMG5Z3B75rnL793KjnNDt+4+EuthuzN/Gop24c611Mj19S5zCp2yj1AwdN1ainEXGTOH5hJub8pgL39LFZ77m519WNveT9dGyQ3w++ZcOfaMdPbrut49cl/dV0IFL4LYSw5HcoIYSQ7qyq40fhtyA2zI5fOLbfByCXxlmF3+BgBWF+1DMsjExWFcPX4RV2BW3sN4GIrC9evGm+v9ti1LO441fseBPRWebCFXv8ZhHuYo/LljHwMlFmnzjPssevXwh32Z841bPatRS8AxZ+cZzUcvQ9d+L9wiCKFyIgslTP/G0PLRHXdmy1OOoZxbryMZBRzPWel7vfe0EEpbJdUyFx/Cbr8UvuR7sdv6kcP84ZLoR8uAuFHyGETAtTPclEbPRc7KapnsUOPyATMW12/KpTNrNgkmJJdFOqZxWJ41cx6hmNC78vXEiE39bAazXqGcX5+gTHUSalE6gf9Szu+EVxheM3ifCrKIsXBj23Ysdv+nAXuY8Dz83F4GfhLu13/MocUptJ0ihngS18pEpCt7z9RQm/rMcvf5zGvfOTcJc2jtjIcglll7PqfSyvpfWem7vM3ijEmu+O7W6eXJ/M8ZP7M0/HT/Yj6fgtBvs1e9B9nYQQchgJY40V1H0UfotivedhN93x2ygRfjKaVyfOmoTMKMxK2ntefkSzqcevCt91qsNdLIfNdRQcBTx3c4iB7+CuU+utRj2LIqXYLydjdGVCppjqGWvkxlSzFNTJRj3rnoMkeCW5vlmGuyiV3Z+B7+TcGLmdti5ZG3f3oEc9RchKgTuQPF9tGEVJIuZBj6aaHb/C+02em747geNndvzcrPOy5gMcYHzUcy+ITJekzcn1Hq7vBa1P8OX+tO3xm27HjwXuiyAs+fCMEEJId+j4kYnY6LvYGSWOX9mOnyRb1rlNcjJfJWTsRMpiZHwQZaJwEjynxvErdN6JML371IYRuk0Ur8N1nMKoZ+r41fX4WY5fWarnJAETxdHTIsn+3WzDXWT/Upycge8W6hxk5LD9jl/THPpBFLjvBxE++8x1AJZY8p0xp7YJ+eDhoN0jebyLdSaZ4+e0HpnNOX4NI9uhcfy83Ht9bxRjUCL8Tqz3EGvgxn67cU9JKW2f6jn562S4oOeMJNhJtAx3IYSQ6eGOH5mI9Z5nCtzLdvxE2NS5cr2mE8Y4NrUKxTTOMO7m+Hk1nWPFN4Ec312n1rHRc3OOWBXF0UrXQWm4S1m3nqvyjl8U67zj13LU87PPD5Lq7QAAIABJREFUXDc9aE1jkgMr1XNW4S5FUW6LS6BDgXuDawkkzqjW8x0D+6+fegavf+vHcHVnZO3FuWO7mU0sqhNObq/oeNsJnW2dU+netIVflZCX+7vRdxFE2UjsXhBirVfm+CVhL20rHeQ91brHr9OoJ4XfIsnXOVD4EULItDDVk0zEZnoSd2VnVLHjJ4Kt+kWl0qL0MsdP6yQ8w6R6Fkc9w+47flXOUFwUfrNy/Kz9LxE7peEu6ddEvGgrZRRoFsrCG952P37hI48lt9cwJrlmBa/Yo3fT7PiN0k44YeC5CGOd9a1NXOdQv6cIWImoc3QDruyMEMUaz17fyweiFGo4mjCF49HBjg02pXpKJ2GbE2tbLMrrq+4DFQBYS2tV5Dj2RhHWS4VfUu/QZs9Pa91uxy+9zYHvTDXquYg0VpJ/bVH4EULI9ERx/VTeskLhtyDW05O47YpUT9+MetY/Rb6rxnaOACsERcJWiqOecVfhV73jF8blQuuuUxtYS8NsmggKY4lyfXKuIverTBDL14zj17HOYXs/xIXr+9l9qh31tHb8RhGOrXnp/ZjO8bPFpozz7Yf5XcLW4S4N46pA9jqb50mhuJYXbwyNu9RLkzAnue1FuUdV4mVkuZee004Y2aOeTa9L4/ilIm9kvQ7KRz1Tx2+nWfjZH1Ds1zjyIlQHvttpVFBuh47fYrA/rKDwI4SQ6YlaJKYvIxR+C2Kjn52wle74md28+hdVzyt3/EwJuJeFxMQ6+59+0FBTUEVdgXsca+O6ybEBwN3pqGfrVE+7zsHNu0GmzqEm3CUyBe4V4S41J59xnDggV9IxubBBIPd9x+z27QWR6VSbbscvf5sS178fRAii2AiLSQrcm0S+fHueJ4XyOJ2/sZ9z/LLnePGjnvc/fhkf/sKl2tstHmdRxE6y49f3nOZwF2vHL7lcOuo5Kg93kUL3NqOe9m3W7b5KUFRbYTv28yxwXyi5cXkKP0IImZo2ienLyLjiIAeCnMQBKB31lB22Jhu55zmlJ8BycugVdgWDKIaC0zmNqK7AvVh2LmLjrtMbWGs56ll8I8n9lylAOQkuE63FHb+kxy/7vuc6cFS9YJATVLPj1xju4poT//0gwrE1P3c9XQgindvx64vjFySF50LbIuYgah71lMd8nieFMhJ74cY+7j61AWDKHb85jA3+q/c9gjCO8c0vPTN+u2G2Oxpb+6P5hNK2O37j4S5y/UXkQw/5sMh2/J5XMup5Ih31vFYy6ikusfx+sV9Dw5oPZmT82HU61jlE8mFMksa6iuMxq4z9O5t1DoQQMj1Fo2JVWD2pekiwxd5m6hLZmDTOxlHPesdPXDfbVZCEN3uPrC1ezY5f0puX/VvG2G4/NjBR9E3jiVGcH3Ms7n/JSXVZuIuTVkjkUz3zb8oqh1SQ711JhV9T7cWan69z2ErHdqtO4ttQ3L80o55BnAvIaSuUorjZ3fUmFF9d2Bslj9MFa9RTkjAnue1ZjQ3+6/c/gl/88GPm33Gs8YUL29gdlgsgu3zcTkmU14yI2IlSPV3HuPJ16bwATJCLLfzKHL9jAw+uo0p3/N78u5/BD/3mp8aOHWhw/KIo7Vzs6PhZ181xz4MnV4nDVE9CCJmaNonpywiF34KwQxnKHb/mOgegzvHLj0TayYFhwQ2chGIthE3RRex5Du68ZR2Oo8z9bRr3LO7UFd2g7H6VH7vdoxbrfKonkJxo1zp+YV74JalNdamejhFje6MIG/3kpHvqHT/P2vHzslFPu8i97W20+eXkHIDwy3b89nOjjpPuF85q1PNDj1zEOx94yvz7qau72B1F2BmV76Lau7Q5p8wa9UxqMZqPK1/n0BDuYnb8xsNdBiWOn1IKJ9b80lHPJy/v4Kmru9l1W8dat+Mno56O0+01Yt83Cr+Dh+EuhBAyW1bV8eOo54KwS9u3SsJdiiOaVVQJmeJIpG85fvI67R7uUjWSlhda3/qyWyGGm4y27o2yPbgyqnb85GTFhLtUPC6uo8woUxznw10AoOe5jbtMQOKm7I2iZNSz5jkYeC6iWCOIYuwHMdZ8t3Yctg3jO37Jyf0wjLAXZF9v3XsXx9jw69/qB+L4pcLiwnZ+x0/u6kHv+I3CGI8/t4Pt/QBbAx8Pn9sGgMqRZPs5td8Dtoh1lEKsk7RMpapfN/aoZ3OBu+z4FRy/UYT1EscPSAJeykY994M497jZznTjjt8Eo6xFgjGhXP07gMweec6UovAjhJBZsKo9fhR+C6LR8WtR5wAkJ45ywrgfRHjwyav46y8+bU4sTZ2DlL2HsRFD3cJdqkVNrPOi7Yde8xLzd+P4Nez5Je5UJm7kWMeEX4ULZ4+iRXr8TdmvcEgF+75d2R0haOhpGVj7dzJ6VzV+25agIPz6xvGL4TlWSfwE4S6NqZ4TVip0wYS7XB8WevzE8Wt324Gpc5juWOUYPvvMDbz6RafwyPkbAJLXaJlwK6biFq9HHD8g/QCj5v01SYF7UAh3GUUxtNbJ663E8QOSSoerO+OO314Q5U78g7aOX5RVjHQRDvZ9Y8DLwSMfVPQ9h6OehBAyA9jjRyZiI7fjVz3qWbbLZmP3+L3nL57F9/7H+3Hu+p7l+BVHPeMs8bOD4+e5ibDSWuMvn76G7/mlPzMnjHWffsgJ6k5DpUNYeCMV++XCFqOekXH8UOL4Nez4WSelV9PeubrnQEbt9oM4SVnsuWPVGZNSDHexxeV+bsevvVBqSp4qhujMA3H8Lu8Mzeugb4mlto7faEaOnwiQzzxzDQDw+fOJ4xfFuvQ1knPK7N04O9ylZUJpvsev3vGT53m975rLDcMYsUZpnQOQBLyU7fjtjaLq+9Ei1dNVXR2/OHMs2eV34IjAH/guoik6RgkhhCQUp9xWBQq/BZEb9ZzC8fNdx4xrXbyxn/45tMRdftQzKQKvDkhpwrdO0j/+2GXc9/hlXNoeAkhOUKuEn5z07TXs+EWFSgh5U8kxZ8defjv2jlVUSPUEZDS2ua8MAC7vjJLaizrHL3VB9kaR6VWznxObIIrxv/ynT+L+xy9XXh9QUuBeEu6SOL1tw13qS+iB8dqMeSCiVWvgmWt7APKpnm1SSrXWMxv1lN28zzyTOH2PpMIPyNxJm1GkzfOSG/W0w11M72T9fbHDXXrpPueo4v4bx8/PRj3lsSwrcAeAk+s+rpXs+O0HUe41bt+Pph4/GfXsWuAuv/O443fwiFjvuXT8CCFkFqzqjh+F34Kw0/hKHT+nfbiLjLxdSUe7ruyMxlw9e9RzVBCFk+BbJ74i+OREsi6mXcbUGkc9izt+xXCXuN6tzDl+uiLVs+Wo53Pp/atzy0SUXd9LHvs134XvlY/DPnbpJj70yEU88OTVyuuTY8gXuGfhLnJyvtX32u/4tQl3aSlYpmEviMyHHE9dSQJG7FTPNrcdxRpysZk5fk9fw+4oxBOXd/C84wMAwE7J69R2rfKOX3JZ31WZiG14bkbpc+w4yqpzqNrxS1xwU/QeZh8AlKV6AkmX39XdEXThMd1LuyDt4zD3o/WO3+SPexBp89xT+B088uFB33dY50AIITNgVXv8Vu+IDwmuo7Dmu3BU+cmbV3DqqrCFzJWdRKiIU2X/vAi2pAB8ilFPR9yJ2BJ+yUloWX2CYHb8GkY9o6gi1VNnFQ32cRRxHYUwSkZRtcbYntYko54X0/tXG+6SPncyVrfmO5U7fo+k4SFNO05V4S77YWRO+DcHXmM1htCmwH3Sccsu7I0i3H066e/7cir8eq7TWiwVLzPNyKDWGqMwxsB38MTlXTz45FVoDXztXSfTYx1/nQZRbFw3+ziGYYy+50AplX1Q0eBeyugkkNWq1BW4e24mEEdRbD5AqdrxO7HewzDM13/IXuCoxPEb+E7jjp/pKezwEhmFsfmAizt+B4/8rhh4LgvcV5TfeeAp/M2f/vCiD4MQkkLHj0zMRt/FRt8rTf9z2zp+1j7ZlV1x/IZWUXO+DzCItBlD7JrqCSQnEmWOX5VIah3uUhAp43UO9eEuyYmpNpcvPn6NdQ6RLfyS0dk6Z3StKPxqdvweTsNDhjWjpkAiAHI7fl426imdgZt9r/WoZ9iqwL39uGVX9oIId51aBwB8+fKucbwmKXC3n59pnCO5nq9+wQkAwLsefBoA8LV3JsJvp6TLL4hirJe4ViL8ALR3/Kxx3qYdv2Tc2DG3MQpjM4pa5fidXE9SM+1Kh2EYQ+v8Yyi3udn3W+34eR0cP62TnUlTR0Hhd+DI67HnOXN19cn8+Mwz1/HoxZtMZSVkCdDpeeYqpnpS+C2Q9Z5Xut8HjI9oVmE7flfT7rnLO6PKVM8wirG9n5wMbvTLTxrr8Kzwiks3E+EnTkGrUc82O34lo57Zjl9e0I4dXzrqKQ7hmPCbYNTzYqtRz+R78tibHb8SAWUcv6D+xDcI8+K3b416ygn/1mCCUc8WBe4m3GWOJ4X7QYQ7Tq7BdRS2hyH6qaCdRHTa45BTCb/0Z78udfje/9nzWPNdvPzsFoDxDyjiWCOINDZ65Y5fz9yX5Llqs+NXFH5VO35hHMNz86Oe8p6rc/yA7HUJ5N+nxV7MrYHX3OPnOXA61DnIY2V2/KL63wFk9uRSPSkcVhLplp0mOIwQMhvk1ygdPzIRG32vdL8PmMzxK5aOX7lpj3qmjp+XjYlJsMbzjq9NfMy2O1F0/MIWo55lI3Q2xVTP4u5Zmx6/MNYmnbIs1bNNjx8AXLqR3L860ZSNelo7fhU9fo8Yx69B+BUK3JMxwmSXTEb3tgZ+Loq/jmRvsl2q57zGwJI0WY2Nnoczm30AWU2FN0GBe1ASqtIFeQ7OHhvgBbesYRjGeNnZLSNO9oL861QeaxFaYcF5zO5L+x0/EXJNPX6Symq/93ZbOn52wIs99im3JY/nZt+rf1+kx+t1EH6Zq5jvISQHRxjHUCrtYaXwW0nk/+9MxSVk8cjKlNshK2PRzFX4KaVeq5T6vFLqUaXUm0u+31dKvTP9/v1KqbvTr79SKfXp9L+/UEr93Xke56LY7LvYqigzF7HRdMJuB4kY4bczMiemcrKYnVxqnLuejDCeTYMsJkGOa3cUmUATcbDqHD85QS0boRPiWCPWebHrFeLxw4ZRT89xEEW245f/fuOOX3r9pzd7ZtSz7jkQx++aNerpl4yTXtkZ4cKN/E5k9THkd/yUUuh7DvbDGMMgglLARs9tfQIetBj1nER8dWHfCiO57Vhe+GWis/mEpm39QBN2j95X3ZGMe7787Jb5gKL4Oi3r0suOIzL3RVJoJ9nxk/dUZbhLGgTTsz682Wtw/E5upI6fVelgJ5WaD2uMKGvh+LndCtxltJw7fosjjJNxYddRDHdZUYzw4/uHkIXTlDexzMytwF0p5QJ4K4BvA/A0gE8qpd6jtf6cdbHvB3BVa/1ipdR3A/gpAN8F4LMA7tVah0qp2wH8hVLq97XW9XbRivHD3/YyaJT/T9htW+DuumbnJ+tJG40ld4qACqIYz17bw+nNfmUHWB0iEEQ8AoVwl4rjdRyFge/U1jmEJW8ktyBI5JPrutsJY20cwqLj12/a8Uu/d9uxAZ54bmfseIrIY3jFcvx6njPWVyhuH9DO8esVFOvAd7EfJMXiA89N+hRb1zk0j3qKtp2f8EvDJXoubj02AHDdCJlJUj1nteNnF8h/5fOP479/5lwi/MTxK4x6iihbN45fcdQzf1+aIvPz46EKSlV/kh+mu7Olo56VPX7i+FnCz3rvyWMnt7k5aHD8cqmek71GzG30k2Oi8Dt4ZM+3ax0HWTwc9SRkeZDfo0z1zPNKAI9qrR/XWo8A/BaA1xcu83oAb0///i4Ar1FKKa31riXyBkCFOlpxXv2iU/jGF50u/Z50xzUWuKd1DvYn+4njV13g/uz1fTzvxORuX3I9yXGdS8dFASvcRdcvum70POzWjHoW9xIBmNFRO9zFr3mjeY5CrLX5VLu0wL2F8Dt7bGAi/dukesoJdtWO38Ppft+ZrX7zjl80nsI58BLhtxckJfF2X2ETYYsC94N0/M4eS157XXb8cv15M3L8XvnCWwAkiZ6S2rlTeJ3KyZYElNiP/SiM0ffz96UpAMUe9VRKVSbBym37jmM+DBiGLVI918Txy0Y990tGPeXx3Op7GIVxpRtkhJ+aXDiMVmTU8+OPPYcfePsnD6UjFqTJsG76+5GsFlpr8//4so5YQsjBEkWr6/jNU/jdAeAp699Pp18rvUwq9K4DOAUASqlvUEo9BOAzAN502Ny+JkTwNTt+yainfBp4x4m10h4/e9Tz2Wt7nfb7kuNJrufZnONnpXrWHO9az8Vuzain9OadTnfAgPExwCgNuqhCPtGuTPVsGPWUx+02awy2TnyP1Tn0ynf8Hjl3A6c3e7jjxFrtqKcEb4wJP98xqZ4Dz4HnqtaOX9iiwH3eO357ZaOefn7Us92On91B1z0kRJ6Dvufga+88iU/8n6/BVz3/BNb75emz8hpfMz1+Ondd/cL7tTnVM/sZIE3nrTihkzoHexdQHMl1v3xoo+c52Op7hVHPcbc0sBw/oNp1HEbdHT9xS5e9wP1jjz6HDz58ETcb9pBXkeR3QLcdTbJ4toeh+Z3DHT9CFk9YcY65CiytR6m1vl9r/VcBfD2Af6GUGrOolFJvVEo9oJR64NKlSwd/kHOkdbiL50BrmKCVF9+6iZvDcMytMsmBYYxz1/Zwe1fHL3Upco5felIfRtrsOJWx3nNr6xwu3EjE5G3HbNGVjgGm/68LGsrIJW5eRu2Kx9NY52A5fvZ1VjHwJNXTDncZF5ePnN/Gy88eQ78hXEZOxMUNMrfjZ47foOfCc9qFNMjeZNtwl3m5HaZ+oOeko56YOBAFmGGdQ5h/nG/dSo5J9tjGRj2j/KhncddQRKzTUsTaqZ4AKgOBAAk8ShI1PUflCtwHvern9cSGXxnuIo9jEGfhLgBK9/yk87DvJh84dA93We4dP3FHi8/9YUA+lHMUhd8qcuVm9gHOsn5wQshRYpV3/OYp/J4B8ALr389Pv1Z6GaWUB+A4gMv2BbTWDwO4CeAVxRvQWv+S1vperfW9Z86cmeGhLx6Txtng1IigO5+KphffugkgE1HFAInLN4fYGUW440Q3x09GUJ+9Pj7qGet6x2+959XWOVxIxeutxzLHT0Y1xfGTT66rkAJ3EYrFlNHGUc9oMuHnuQ58V1kF7uM9fmEU4wsXtvHys1vo+25jeiIw/rz3fRf7YYz9UYQ1Px31bPHJr6RR1rmkwPwL3I1Q8V0j7Iujnq0cvxnVOdg7fjZKKaz7bsmoZ76SoDhy2is4fo3CL8oLv55X3v0oty2vB3n97o0iOApju6A2x9d8E8AElO/4yeNZF7wi973nOYlwmHBUcFQQfst64irVF4dR+AVR8ru5i2NLFs8Vy7nnjh8hi0fOSevMjmVlnsLvkwBeopR6oVKqB+C7AbyncJn3APi+9O/fAeBDWmud/owHAEqpuwC8HMATczzWpSNz/Jp3/ADg/PVy4WdGRtM/n7yyCwC4veuopyuO3z6Or/mmZgBI6xxqjne959bWOVwUx29rXHSZOoeoXlyKI5GFu+S/3/OctO6hepcJyIvPJtE08DIn0+z4WWN7T1zexTCM8fLbU8evRvwGBScquw0nc/x8F67bbteq7adSjtmlnM9JRdmoZxaIkvzZ5v4EhVCVrsioZ/FxBoD1vlvp+EmYSpXjN1GBu2s7ftUjyEmPX7arG6Spnus9D6qiPgVI9hFvWiFD+6Nxx0/+57VV4/iNLBe6W51DcvmB78J11NL2+MmHN3VTCatKGCWvIdeZXLiTxWM7fhR+hCweOn4lpDt5PwjgAwAeBvDbWuuHlFI/rpT69vRibwNwSin1KIAfBiCVD/8DkiTPTwP4rwD+V631c/M61mVkkgJ3IBF6SgH3nN4AkAlB4xSk1/fly4nw6xruIiLo2et7uHWrnxtdjGM9Vp9gs95za+scLm4P0fccHFvL9paKwR9hyf6bjaPyO35jo55WJH4ZozQ6394zbBqT7FvJin3PyVVsAFmi58vPbqHfWCCfD+URBr6LYRBhP0gcP7/lqKdcX2OdgyvCr/EqO7FvhZGIsDd1Dmacd8IdvxmMevbLhF/PM6PS5vIS7tIfL3BPevzy7mXTfSmOevZKAoGEwPqwQ3ZU5QOAOjb7Hm7uZ8Kv1PGTHr8ax8+MxbrJuGnb3dLs+LN946ZR60UiY7HFDsfDgCTD0vFbTWSHH+COHyHLwCrv+M2tzgEAtNbvBfDewtd+zPr7PoDvLPm5Xwfw6/M8tmWn7Y6fCIQLN/ZxYs3H6a1EsFxMxyYlAVME4JNXkoqCrqOeIiD3gxhntvq4uD3MFbjXiaS1nldb53Dhxj5uOzbIuRjFMUCJJa/CpHpKj19x1NNKRiw7cRYnRnrQgBaOX+r2DPzkxLjo3jxybhuuo/DiWzfR9+pHPYuhPPZt7AcxHEfhlg3XnMBprWtdH3nc6sQyMFmXXhdsx+/Euo+e52TCT7VzyYD8DuQsCtzLhd+4My1iRXr88o5fZF5XEzl+Xt7xq+vxk9vtuckHLVon+5J1bPS93MhqufDLVy2UOn7GhXbN+2sS7H3Kvr+8wu9wO37pqCd3/FYSe9RzWd8/hBwlMsdvaaNSKpmr8CPdEcHWVOcgJ67nbwxxy0YPp1LBcv76PlxHGcdLusIu3BiOOVqTYIugM8bxy3r8ivUJNhs9t7bO4cKNfdy6lT+uYi9akH5yXYXrJP12dameQPX/PIN09+qWdUv4NbyxZfxP/izu+F24sY8zaW9i33dqUz2rdvwGvov9MIKjlEkOBcQBrX48ZA+w6QMEEV/zinq3hZ9SCl9/90m85Lat3LG1GTM1DlW/vneuiWHFSC1QHkI0Hu6SHzmVUc+2tRijSOeFn1cX7pK95pNdQI0gjCsTPYXNgZfrk7THV+W2giiGl3Zsyn0ZO1ZLJHfpgbNf0yJcl40kLv/whrtIQBAL3FcT2/GrmgwghBwcYctpqmWEwm9JkRPJMkfCRpyG89f38OJbN3Fs4CephEGU+1mlFHwncUnOHh90Xki1RdCZzX4iZAKrzqFGhKw1pHpevDHEX7n9WO5rxZTEqEWPX27HryTVE6gZ9QyT8Ji1nos138VeELVw/PLCr9jjtxtERjAkO34tUj0revwcpTDwHLNLGUYadRN/oXH8WtY5zOmkQk6mB+nj8I4feJX53iTBMoE1cjmPcBcgcabtUBT7dtdMgXuhx0/cy/Rpa1Pn0HbHLxn1zGpZRmGU1HpUdPgJm/3Cjl847vhJVYS8hst3/LJ9yC7CQZxM33Uaw5UWxV4QmeOqm0pYVeQDIha4ryZ54bd87x9CjhpyjskdPzIzbjs2wM9+z9fgda84W3s5GeG7uhvg5HoPjqNwMnWriuJBTv67BrvY1wEkjt/Ac80JZRTXVy2Ik6IrXKWL28NcqApgiQKz41c/6pmEnsSQc5uycBcAlWN1dtriLal7Wic0AWvUs5cJP+njA5L9tjUj/BpGPcOqHb9k1FMK3DPHr/4kQB63Jtcy2/Gbf4F7EXeC2za7dj1vpnUONhulo57JsZWPemavGRHkjTt+UZz7YMav2X0Lo6yH0U71XPMbRj17HvaD2IjUsnCXUVoOL8dS9qGE7Y62rRGxsVNBe56D4RKeuNon1od11FMmQFjgvnpc2Rlho6RKhhCyGMyOX8OH6ssIhd//z957R92S3dWB+1S48Ysvpw5qNd2SWgEFJGEYGwmEZNmEAePBgM1izIKZZZhhAbaZZY8xMhMYr/EsPEQvGbDBMgYsgohiCJKQBAqt0FJ3qyV1fP36xS/eWHH+OPU7derUqapTN3zhvdpr9er33ne/e+veqnvv2Wfv395HGF//qgtY7bilt5EXrkRUyO7pKota+vus831AlpCcXlUUvzjOzdTJ6LUchFGsVTYG0wCDaZDp8AOktEmyeobl1kZS/ITVU1PnAJQrfnSbzT5/7auk/Jzi57DkWPljjLys4ueFUXGqaKKuqOdO9PhJdQ5AtUJnWudgK6/zojH2Q9jJ/KMK0woEINsJN9+MX1rgrqKrCSFKCWfW6umHEcIoRidRDs0L3HXhLmVWz3RW1w9jvgFQEe5CQTQUVKOb8QvCGK5jpYqfxoYsk2SLzaD4HYNwF7nv8HYgfltDD5+9siv+7iepnk2B+/HEraGHs+v8u/EoWqUbNLjTQKMpjeLX4MAhLx4pkIQIi3pB0qL7/PpsiZ6AbsbPTsImYiPFD9DP0FCVQ27GT1GD5Gh7HeyqVE+pyF4HOWb/RJ8fS5VNUiV+qp105IfoJkoRWXiLiWecuQ8C9f/xeTIbtm1WgWA6gFynS28WTPyokKiYBqIAqVLbby9I8dNcS31NCJEvWUO5XS5R0aR+QsCsFiMIuSKdtXqyklTPSPRnCsUvqXMoA/Xm0Zzf2I/SLj1lxq9M8RMzfkmB+zwzflU9loeFbSk8Q2d3PW74v/7wcXznL3xU/D1MrJ6Uerxo/OIHn8K3/NyHFn6/DTi2h55IQ24UvwYNDh/HecavIX7HHLKCclIofu3cz4B0oXlhHsXPUhS/JNylKExFBhE/NSof4KEzAHKKXz7Vs7zHj2aQClM9aYEbcKXmH/7SR/GXT94SP/dlq2cvIdAVATuC+ElWTyAlC2MvQM9NrZ6AfoFNj8+PUw13SY+B1zmYWT3p/irrHKSZwWWgrH5AqI016ikWMeNH9QQqepoQIqFaOQyOVGkwSc4jnR+TWgy5F49QVuBOc3j8djamIbd6VtU59FXi54VY62RL1LmCXqH4hVnFr+7mQCbVM5lRPGrYzih+x7/O4WPPbGNPmlP1k47VZYW7PHFtH4+9sL/w+23AsTX0cC7ZsC0aU2jQoMHB4Tineh6/I26QgWxVo9k+MZumKFW0eJy1w0++DyAJd0l6/MjFW7L+AAAgAElEQVQiWEYwSPXSlbhf30/K25UZP5UUyAmHRccnK35lqZ7X9yf4k8ev42NPb4mf8x4/snry17FKyu84VOegEL+EHKhWTwCFyZ6FdQ5SCEnXtYzDWNI6h4oC9+ThljX/w+cc9R83tRS/SJrxm8fq6Ufa+T6AE/iJH2UITs6uGGYVP+pyNKnF0M0Xmha4t2wGP6BZz/KPb1L3KOBl4odY67qZ5+Mn84N0XU5KFD9R4F7zGpEDi45quMvO6PaZ8dub+PjijQG8MJI2zLhq7Cwp3IW7EY7363ZUMQ3CzBhE0+PXoMHh4zj3+DXE75hDN+OXEj813GUBip/UV7bZa3Grp7RILiNJNB+lW1hdTxS/M4riJ5QoaQFTtsNC/Xa0tlFHDtvSjB+pjOrsEymjpKBWzceR0iesnhQgE5LiJ4e7FMfmy7+jK3CX/+waWj2NC9wts/ubFWUzaYxRsbRBnUNAit+cVs8wLEzM7dMGhXxdhGnojmOnih8tdun8mKiXOuJXNeOXsXomil+V1TNV/NIZv/WE+IkZvyjKKH66xbsc7iL3R5oiQ5rn7F9cFraHXB072W8d+zqHRy7vgk4PbUyQaryscJdpEMGXanQaLA50bZ5LNkWbOocGDQ4fJmveo4qG+B1zyASBFKqTK0RYlkf8Tq3wBNG2a2FiaPXslhC/a3sTdFwLq+3sYlYoUZLiVx7uwlMHowIFsmXzY/CCCNeSuUJZ5ZD71e4/s4qVtoO1ioCdfJ0DS+5LE+4iFthFqaLFqZ6Ebss27r4LCoikCnqdl7VwqwojMY2Z90Oe6tpx57R6Vih+ADCSqhDkLjvHtqQZv8Tq6aQbIkD56zjVzBfyAnf97/BFe1rnMA3CUussgcJdBpLVc1WxenoBv286llLFz06V5jqXiS9d0+0jqvhtjzysth2sdvLzncvCYy/s4Sf+8PFaJNoEn3xuR/xZEL+kx29Z4S7p9XT0zu1xx61hdlO0eY0bNDh8NIpfg0ODvHg9qSh+LVslPcyIyJSBiuBPJyEsHUXxK5/x44tO3QzNtf0pzq51wBSJLq/4lQfIkCJRmeoZRCJQZqIofkSS3vrQWXz0n32NUE6KQIt+Igy0iPaT9E6Z9FRaPQtCR/KKHyWHmoa7mCl+SyN+FTNpjuHsEVkTSTkqWzR/9soufujXPqV9TmqdggwiTCNN4blrW3CtNIhFDXcxqcXQzfiVFbjzREYmHn9/wt8/VamearjLJAkZkisVgihCK1GCWsm8bu54FcWPfs8U9Pt03o5quMtG30W35RyY1fOPH72Gn/3zLy6caH7i2ZT4jQXxSxQ/xhDFWDjZpHPa2D0XD1mN5iFQR+/906DBnQbadG+IX4MDh6x+bVZYPR3bmmu+TzymZeH0Cid+bZcvFk12P3qlVs+JSC2TkSoMSXx+VaonWT2LUj2F1TMssHqmFkDGmCBzZSAVLzfjF8QiLEOkeppaPSvCXWxDouYT8aua8Ut+vCyr58QPS19LU8WPZjBly24R3vfEDfzXhy/nytiBCsXPpQ2KLPGzGD9OVwpiScNd1Bk/A6unWuBe1OMXpYFGLccSoR29imtTWD09SvUM0UkCVkhdpJh/gF+b2h6/ME/86mwQEFlnjB3ZOgfqQe217AOzei5DJYvjGJ98bkdsCtD1yUOxZjt/JqDAnqNI6o87SPE7udI6su+fBg3uNASGm+pHEQ3xO+Zo26nKRDN0lOqpLva/4v5TeOtD5YXwJnBsJhQ/CneJ5iV++1OcVoJd5PsTBe5hOu+kg1D8iqyekuJHVk95setLVk9TdFWrp0RKaBHZkwrc1ceUYRLu0nFtcW6rdn/Trpny51Rnzm4WmFg9TXv8XMmaWLYIGiTKWKB5jbjipz+eVPELMrenc5JN9STFL7F6suxGhQ5FM346Eks1KXT+Wo4lbJamih8phOOEfLuOJfoi5V7MjmuXKn5t267VuSj/Pp2voxzustlroevaB2b1pNd6ka/H8ztj3BxM8bp7NwFkrZ6uzVLityzFr+BzrcHs2B7y4KHNXiuz6dSgQYPDg4nL7aiiIX7HHHLZONkkixS/H3zLA/ihr31w7sf877/iRfj6V10EANHjZ7L7Iaye06zVM45jXCtQ/OjuiJCEUpm1Dk7SsUbrUvVwZMJwbV8f7lKVgKkirXNIS7YBTlKI5IpwF7fc6lk049eWFvlygXul4leja4aTr8qbzYQq4meaNhgkREUm8EWg2TZfc7/ToDjcRbdB4QexuHZcKYhlooS7VNVixHEsFEg11VO3oEvn41judzoVil87UejkOoeua2dUA18itB3Xqkz1tGpUb6TPIRKbIW1nvtnMZWF75GGz56Lbsg/M6ukJe+TiXg+a73vjfScBZMNdbIstUfFrrJ7LwtbQA2PARq+VpP824S4NGhw26Dv+ONY5lA8vNTjyoAUhVTkAwEbSP1cV6DErfvitKXlsOxbCKBZf/JYaoylBLKiVHfXBNMDIC3NVDgBXouQIeSqcLoJt8RkWIorq8cg9ftd28zN+co+fKUjtUQvc/TASpDJf51BU4B5lbqc+BsAJ5mCarYwoQlBAJHVwlqn4eVEpUbEthtBgQUPKW8vA6lmq+AUmVk9Z8QvF7R1pziYNd0msniUzfv/qdx/Fr37kWdFjuSrN2rq2lVy32RlWmqWTw10IvQrFjzGGfsvGcBogimJMgwgd1046AyUF3U5JWdWMn8kMY+73pcc4qjN+O0MfG70WYnjaupllgK7dhRK/Z3fQciy8+q4NAOn1SQS/TmdmHUyXQGIbcGyNPGx0XdgWK03/bdCgwcFBKH41hYKjgIb4HXM4tgWLpUmeAF9Ernfd2srVLCAFi+aIymbJ2g4/VnWG5vq+vrydYElqUGWPX7JopsWqqnRl6hyS7sC84leX+BXM+EmKX87qOUedQ9tJrZ5VIRspcTBQ/NhyOr6AZMavVPGzDFM9ufJmYvXcJ8VPQyinQSSskCq04S5BSl5cOz3WIqun7rm874kbOL/Rxbd+2V148ekVvPLiuviZXAFiW+nr5IdZJV0mfibzpyttB4NpKM2a2hm7pbyRUqj4haFQi2ad8ctYPZNQHjXI6bDgBRH2pwE2ey1M/PDAFL9lBKJ88rkdvPzCmthUoM+2MJkTFTPTC+YOjeK3PGwNvTSw7YhapRs0uNNAYkQz49fgUNByrIziBwD3nurj1EpeQVs0iMjQYqlM8WOMoadJzaNZuzOr+uN1JDWoqsePwlzIDpNT/JIF6P4kwM6IW+7kxe50BsVPzPi1ssTPC2KhGpGKJBS/gjkiOUREhkz8uq3U6llFloLQ/MPJts2SNesijmOjGT+TfjE/UBS/kkUQWRx15NiozkEJd6HAHdeSw130Be665zKaBnj1XRv47v/mPrzpJWcywUNqBQhBrePIWD0rFD8AWOk4GE4DsdnSTTogp7LV06lW/Oh9IxSjmj1+9PxMQnkOGjvjZIaqz62eBzXjt+hwFz+M8Mjzu/jSuzbFRgRdn34Uw5Zm/OqkspqArpvjOuO3M/Lw7e/8S/FddJRwa+BJ4xtNqmeDBkcBx7nOoVH8bgOsdVycX8+qZf/xu16fS4ZcBmghR4vsKr8zn6HJWqmKytsJ610XO8lcVHWPX7niZ1ncOnp5eyz+jRZHcRzDDyO0ayp+bdXq6aQzfrlwFzHjV9Tjp1ccO9KCnxbvQPEsGUHUORhaPZeh+FGx8yJSPYmAtSoss0A646d7jUrDXTS1I5lwF5vl5rNooS0IueYxh1Kfowqh+CnPR8zOSnUOhKpUT4Anew69QJCZrrB6EvFLw5LKZvzo+NSwJRPIc4RtibAXvf4HDdoA2ui10GtNDjzVc1H2yKu7E0yDCA+eWxGbAqLOIYzgWpbYbFh0uMsy5hUPEk9cG+CDX7iFR6/sFTpPDgvbIw8vOtUHUDwL3KBBg4NFGFJwXkP8GhwCfuW735BTy9Z7s3f11QEtMEgdqdr96GnCE2iXVTfjB3BCSLfhIQVldQ6pZQ7QK5Atx8JzWyMAwLm1jpR8FyOO689GPnB2FV961wZeen4NgGr15Lfpmlo9pRARGWqPn2mBux+Zfzjxjq/FEz+yGRaFqQAwThT1EsW3zoyfbqE09cOSGT+94ieHu8jdeIylSrJVcl7GXohegb00vWayr7+w/kqpnupxloFbPYPUktqy0ZbCXYIMKbO11RdemCd+tVM9nezxT4MIq8b3sFxQauKJXgu9loMgmVmuq/zXBV27i1L86Bz32474vJj6IaIoRhTzzQNnSVbP497jF4iNkKNHqraGHl57zwkAaMJdGjQ4IjjOil9j9bwN8MDZVWwoVs+DAi3mzYlf3up5fX+KrmsXzlydXW0L4udH5ambecUvf5uWY+HyNid+d5/siV1xXcy+CU6ttPFb/+grcGGjCyBL/GSLHWBQ4C5Z72R0pOAY22ILL3Cn29RRckwxUZJN53nsIJnxa9upclSEfWH1rFfgblkMHdfKXKcyEeB2q3TGr+PYmXk1OYxI/n0vjETligr5mlGfL5AqfhniZ6L4tcjqye+369pwHZaSjjBNyS1S/Kay1XMGxSgT7mJw3srw6cs7+Lr/9y8E8V4EtoXi56ZK2QGofotWyWRVl66NsR9mEpfTGdTFEpzjrvhR8u+iQ2/mRRTF2B75ONHnG7l8xu94kusGDW4npGur40ejjt8RNzhSIOsi2eKqCEZPY/XcHno4udIqDHs4u9bBtb0pokSRK3uj2Qrx0yp+toWbA77Lf+/JXjoHE85G/FSIGb8wnfHLpXqW9PjpiC2RPREiYlrgXiNy2LbNuvTqQl6QFj52nR4/h1VaZgFJ8dPcpmzGD+CESb5OfYm88CCaNNVTTlyl56KSTbEB0CpS/Jj2+ZSlehoRv7aD4TTMWj1lxS+K0BLzd8Uzfu05FD9f6fGj+5wFH/riLTzy/C6euTWa6fd12B7RjF9LvE8PYs5v0TN+8iYTWcMnfpS5hpYR7hJGsbjej+uMHyn0uuqXw8TACxBGMTa6SbiLbVVu9jVo0GD5aBS/BncsyLo4nCbhLjNYPXfGPta7xdbUs2tt7I59DAySQ4XiF+pn/IB08dmyrcTqyVMGaQE2bw2GqHMIIoyThRB1GDLGSiPti2b8AF4ZQIt9ep5V1iSyMJmmei569gcwI36m84VpgTu/r6JFcyBVaWh7/Epm/ADk+tw8iZA7drbAXQ1Z0VVTUOptkeLXKlD8RI+fpVH8jKyeNgZTacavZWVn/IKoUvGT1c5ZCtzlYBx6zWcNdyGlnsjaIiCIX8+VOhyXX+mQ1jkshmSOpaAhx7bg2gxjP8wkwy4j3EV+Dx5Xq6cvhYcdJahuiSbcpUGDo4GmwL3BHYvU6llD8ZsqxG/kie5BHWjY/vkkkKXsMSwTxS855jNrbaHATINIkLG5Fb9MuEsAxrI9fG3HKrF66mf8AL6gE0Xhhn1qajhIGUwDVuqClIiqHj+T+UKyDVYpR0PpGlMXc0TyKxU/6T7kgJKWbYlF+yTpxlOfi0qgRa1H5YxfkdUzq/g5FjPaoOCKX5rq2UmCgUSdQ6T0+GmUruyMn5nSLCPz2s2p+FEo09ZwccRvZ+Sj5VhcKdPMdy4Li1b8JsoGS8exMfHDjN2bPh8XOcubJX7Hk5QEgvgdLTWNyLxcFdTUOTRocPigtdIx5H0N8WswH1TFr2r3Y73r5gIkdsa+sLLokCN+JQteVQnTKZC0eD671hGEbOyF4nfKQkhMoPb4dd3sDBi31BUofkFYrPi5lljUkXWzypoU1LF6Gpao14Wp1dNk0UUhK2m4i36Bvj/1pd9R5u0MznO3ZWOk9DvS7VXFT70fR2NbFZbfgteA5jpzip/Sw0i3M7F5Apz4BVGMveQ9R6memToHsnq6Fiaa6zJT55A81XkK3IHZlSEifjuLVPyGHjZ7blI3c/BWz0WRJVJr6drotGxu9RSqvyUptgt5SADZc3lsiV9E1mf9df30zSHe/8SNgzwkANI5FYnR1pGqQmnQ4E5FGPEO3KPSR1sHDfFrMBdo1mrs84VtFfHb7LWwNfIQSzvOuyO/NIVUEL8dvugrC3dRZ/xszZuSFupn19riC3UShOILdV6rZ2o3jTHy8xH+bccqmfGLC5Wojqz40QKuYhEQRhGYphdQB9uylmL1VJUI/WObz/g5NqtUjgZS+Idqa6PFaRnx48q0POMn1znIM356q6e6gKSNkV673OrpBdnfC4TVM6v4mdg8AYjApBsDXpnSbdloJ1bPUEmxbTs2vCDKvDf5MUXifU6KXx1l2AvCjL0amI0gxHEsrJ5bw3z66KzYHvmiB1UQvwNQ/KYLVvzUDRZu3Q3F5pBrM+GAWKTVUz6XRf2kRx3C6lnwuvz8+7+I7//PnzjIQwKAjEUboBm/hvg1aHDYCKL4WNo8gYb4NZgTnZqK32a/BS9I56/iOMbu2MdGxYwfkBK/sscgZcsTil/+NrTQPbPayaT4iVTPOYkfY0x8QY+9MKfOtN0yq2dxamnHtdK+ONuswN2PYuOeGZ1SpWLshdib1Ft0i0TJ0lRPy2gxGpDVsyIdkoJd6HdkeEbEz1HqHLLJlEKtKQh3UZVT2hjpF4S7yN2PMtQZzdYMih8A3NhPiB9ZPcNIPBbdd6cgMMcLNQXutayeqX2ZCOQsZOfW0BMKyCJn/HZGniB+XZc6HA9A8Vv0jJ+SINx1E6tnSLMoywl3md4GVk8R7lLgOtge+tgd+2IT66Agalic1OrpB0fLjtqgwZ2IMDRfWx01GK1wGWN9xpiV/PkBxtjXM8YOpiiuwZGGmuqpU9hkbCbKHs3oDD0eN14247feddF2LGH1dGukeuqOpyUUv5T4Tfx0IayrU6gL12bwgwgjL8ipM+VWz+Jwl6975QW8/RXnAaQEt4r4BUnvnQksgxm/f/abj+Dv/tyHc6pQGRaZ6knhN1UF7vuKWicjVfyKj6fXsjN2P/m8yEE0kyCv+Dka5ZQ2RvoFil/RjJ+s1vBjrqv48dvdTBS/jpTqKVJsJcUPyCcz6grc68/4scxjzUL86P0PLJb4bY08bCZx+WkNwgGEuyxJ8aPP5I7Lr2GyC7tSj98ilf3bYcavKtyFxhPofXRQEDN+rcbq2aDBUcKdoPi9H0CHMXYRwHsB/H0Av7Ssg2pwfFC3x4921neS7iya1Smb8WOM4exaRyh+i0r1PLvWTmf8/DANd5lT8QM4eRQzforK0y5J9fTD4tCR7/0bL8Y/+PJ7AUiKX1WqZxQbBbsA/LWLKhb0z++M8fjVfXzkqS2j+wTyAQVFj61bjP78+76I/+P3HxN/5zN+TFx3RYugjOKnPCeyo5WGu7TtTFccPy+U6mllUz2dvNWzaMavuM5BT4iE4peQd7pdXcXv5mAKxvi1x7vAokzaI5AqfhNFgeLEzxbPDZg93KVdQdjLQPN9XddeeLjLhmL1PMhwl8XN+IXiHAP8/Tbxw8ycryXO3yKtnqH2z8cJ9D4r2vgilwMp5weFqVbxa4hfgwaHjTCKS/MmjjJMj5rFcTwC8E0AfiaO428B8NDyDqvBcYEId0kWSlUkY7PPF1i0Y08EsGzGD+AkzcTqST+jRa1u8FYOdyHlZCrFns+b6gkk6WthjLEX5gI9+IxfcaqnyYyhozzPIgQ17Ah8Nq18UUG1BO/6yLNG9wmYFbgXhbu874kb+LPPXRd/94PIzOopz/gp5NAo3MV1MnNest3RtRn8iM/Cmfb4CcWv4DUoIrKCnKlWT0PFLyV+nggZajkWgijOKdxFit/cBe7BYlI9ab7voQtr4nNjXkRRnFg9FcXvQK2eiyN+cpAUV/zkHj8mWXUX8pAAFMXvmPb40fu1KGCKFL+DJn7pjF9C/BzWKH4NGhwBhPHtr/gxxtiXA/h2AL+X/JvZyqPBbQ1ayFEQRrXVkxM/2rGnL9SyHj8AOLPWEV+6ZcQotXoWK5A6q+fYX9yMH90HKX65cBe32OpZNuMngzFmZI8MauxK2az6/ojA/MEjV41VF6H4lRCtoucynAaZagY/jOE6XLlwLGY046eSY1qclhH8XsvG0AuEpdWXyItrW4hjvuNXFO6iKqf0GvQKFL+2qyddgbDpZYmTek0VYUVS/Igs0n2RoknW6ULFL9T1+JkvPuXAonl6/C5vj7HedXH3id7CFL+xHyKKgdVOQvzcgyF+YRSL631RxG+cED9CJ9lgCiRldxbFtgq3w4xfVbjLnrB6Lk5pNgHNtNJ7swl3adDgaOC2n/ED8AMA/hcAvxnH8WcZY/cB+LPlHVaD4wLbYnBtVsPqyRdYqdWT/79sxg8Azq52xJ/L3mzC6mk049cWO6kTP0qJ36Jm/MKI78LrUj1nKHBX4VhMzO8Ugc/4GVo97WriN5gGePXdG/DCCO9++LLR/U78EC3bKiWgRcRvMA2ETTKO48zrQ5ZFHfZLUj1Txa9kxq9tI4oh1R7EQhmTg3V0xM/RKKfDaQDHYoXXVmqBzBKOQLFjunaq5piAFL+dkS9+R7VnU+9kp2TGj35HFIDXqP2Q1dJ5Fb9Lm11s9lsLm/ET/YqtlBS7NstUeSwD8vNf2Iyfl+2U7CZzqqniZy2F+N0OBe5l4S5RFIvPkwNX/JTAnpZtIYqPXtF8gwZ3Gm77Gb84jt8Xx/HXx3H8E0nIy804jv+nJR9bg2OCtmMLC6BJjx+QKn474+oZPwA4t94WfzZS/EpSPduOja5rY6XtiMXu2E97/BZl9SxU/EoL3CNjxdEx6N2rM+NnoiAOpwFec/cmXnP3Bt71kWeNQl7GfpizQ6pwNPZI/nihUPzo566V2h6LlKPhNBAkaSbFT1J+VMJJCpkfRphIVQcES6Oc6q4DGUSsJjnFL9vD2LbTxEYTrEgKY1cKiABSxU/cd+mMX5b4mRaAk7K1GKvnmBO/nouRFy4kYVFdWNOfl634LYMsTZT3GRW40/XvyorfAsNdaHNkte0cf8VP83myPw1AL9eNwWSux7k5mNZSq3MF7qLvs0n2bNDgMEE9fscRpqme72KMrTHG+gA+A+BRxtg/Xu6hNTgu6LiWseLn2BbWu64IdTFW/NZSxa+0zsGuVvy+7fV34x3f8BAYY6m9TbJ6mlgtq+DaFrwgxsgLcva+tmMXzsKUpXqq4H1yBsTPMNXTZvqAFUIUxRh5IfptB9/2hnvw5I0hPvbMduX96lSx3GNbVqHi54VcjRWddlInXJnVkwI7VGsUlb6X1jkkStnQC8Rr3LIp3CW9xrwgyoW76JTTkRcI9U0HMV+XU/zmrXPIEhog3TgRip+dhoEAGsUvzM/4mfb4pXOE2VTPumSHd/iNcXGjJ+aEFzHnN0rSO+X3aK/lLJ34TcNsYuwiMFbcBd0WJ7B0LdoS8asKcaoDej+tdpxjPONXHO5CNk9gfsXvB371k/jnv/WI8e1pFpw+q0QIVKP4NWhwqAiiWIRlHTeYShsvi+N4D8A3AvgDAC8CT/Zs0EAUPwMwIhmbPRdbyaJtd+yj7ViVxOCMbPUsLXCnL0b+BW5piN8rLq3jW153F4A0Jnvih5guUvFLUj3VxRhAPX5FM37FBe4qdJZCFXWsnkUBKwSyv620bfyNB04DAB69sld5v7ouQxW6DsE4joWSPPZCsdgxsXoOpgHWOg4slrclms74icdVLMDpjJw+rVQb7lKh+Lk2A2P5GSmqc6BrnjYlTImfY6fdj7JdDEgDcNSqCFlJixLFjh6/rlVQrYyg51mX7GyPfIz9EJc2uzjRywZEzQPV6gnw1/YgrZ6LDnchtF0Lk0Dua7TERpgpcTcBvZ9WO+6xtXoGJTN+uwskfld2x7XmBMktQYE9LeFiaIhfgwaHibBGR/JRg+kK1016+74RwO/EcewDaLwGDQBklRMTcWmz3xKK3+7Ir1T7gLTEHaiweiZfkBR5XbUjQ2rNxA/F7ywm3IVh7HGbVb7Hr8TqWUvxKydqQL1wl6oZP7IG9tsOTq200HIsXNkZF96eoIZO6GDbebI08kJhsRp6gUQiUvVrWrAA2p8GWOm4cGwrNwdpkupJRGDopRZgVyIv/DH4gjCX6qmzek7zyq8Mxhg6mn5HUvzIXurYFr7y/lN49V2bhfelggJeOorVk2Ync4qfdAzCXit1GAJ1iF/29xljaNnF560IlOh5abMrlNztBQS8jDWJs9zqudwev6XM+CnKete1hSoN8Ot2KTN+yblc694OVs9ixe/cWgc35uzx258EtV6jiR9lPjvnsUo3aNBgceAzfrd3ncPPA3gaQB/A+xlj9wCo3upvcEdAVk7MFL80nGFn7FXO9wFZq2fZLos842cyeEuLobGfKkqLmvGjneL8jF9JgXsYCVtcFRzLwOpZQ/GzKqyepBCttB0wxnBhPe1WLMPYj6qtnozlkiLlHr2RRPyE4ldq9fSx2nbgalRMM8XPEY+rKo10jVNyqE7xqzvjByRKsK9aPbOKHwD8yne/AW97+bnS+5JBFtOuq1csnRLFLxQzhvw2FptN8ZM3M8qU2iJQh9+lzR5OJFbPrSUpfr0kFMUE/+p3H8WPveeztR9XtuotSiUbe1niR3+W56+XkuqZUfyOJyEpC3ehDr8Xn+nj5r5nNNdchL2xX1jlo4NK5ul9dNCK3+ev7WN/spgKlQYNbgfc9opfHMf/No7ji3Ecvz3meAbAm5Z8bA2OCdrSF1NVnQOQEL9hmupZ1eEH8MXrarKALSOX8vyVybFwpcXKpnouosBdIn66VE8viLQLiFrhLjZbeIG7keKXkKILG10jxW/iGSh+Gnuk3MU3nIbwg6x61K6weq60naRsXe2lo7mZklRPyeop+h1J8UsI0r4gftnzpZ/xC0tn/Oj5qOEufpQnTnVB56ubS/Xkx99SFD853IUenwgDXUumxEGXlFuWalsEUvwuJuEuALd/zgt6DXJWT8MZvw9/8RY+bjDnqkDCnlUAACAASURBVCKj+C1oEa9aPenPtEHh2hboMlpkuItQ/DpOLVJzlECWal1NCX2O3396BWM/FJ21dTENQkwlBdYE6jk9LOL3zT/7IbzzA08d6GM2aHCUcdunejLG1hlj/4Yx9rHkv/8bXP1r0CBjmbMNSMZmzxWK3+7Yr+zwI5xJ7J7lM34p8TNV4Sn23A8jMFYdUGOCUsXPpYCL7Jd3GMWIYvNFvo4sqahT4G5VzPgNJKsnQMSvOuVON+eoQkc65f6+4TRV3uSgk7Jwl5WOk5StK4pf8jtqGqeMtU5aO0IWYFJiKVWUXg813MViuhm/oPI14EpweZ3DLCCrp5rqORCKX1YJlK+BUHn8ujNiqVqaHr88E2yKy9tjrHYcrHfdpVg9ZRtunVTPraGX2aAwBT3/lbaz2HCXjOKXbFCI9FYmrEmLDHchstc/xqmetDmkflYAwN6Yv373n1kBMPucH20U1bN6hpmNVRHuEhzcpE0YxdibBLg5p821QYPbCbd9qieAXwCwD+DvJv/tAfjFZR1Ug+OFdsbqaUD8+i0Rx74z8rFhSPzI7lmWukmP74eRNthFh3YSe+4FXG1jhr9XhpbDhF2s6+ZTPQFNkIfGFlcG17IMZvwi41RPx2KlMf1ExFYk4ndtf1K5+zzrjF9G8fPSPjK5E65ILdknxc/SKX7Vyu75jQ4Y44RDWICTKgUiSrSg1vX45Wf8QvQriV9eCaNjn2czgpI96TjzM37Z4Bb5fIo6CSXV07TOQQ13ocefhfhd2uyJ319tOwspcS+yepoofnEcY2vkYTStrwDR81/tLI4sTfwos7nQURQ/x1pSuEsYiYCu40v8iuscdsc+LAbcc5Lvdc9K/GhWsI61l8/4ZdVy4GBTPelaHc6wwdGgwe2KILzNFT8AL47j+EfjOH4y+e/HANy3zANrcHwgL3xNyNZmL41j3xl7RuEuQEr8ygZqMzN+hgSu20qIXxgtZL4PyJI3XY8fkF8A6NSRMtgFqZ4jL8D1fa7E1evxK58ZTMNd+PO5uNFBHAPX9spVP3X2SAfHYjkVIjfjp1g9i2b84jjGYBpgteNoA3Dod8rCXdqOjbOrHTy3PcrVfNDrSTMvqnKoq6bQ1Xqo0C2c/ShOkjDnIX5Zq6c646eG1sjHrs741S1wV88ZUD6bWYTnkw4/wkY/rYSZB2JzJmP1dIxm/IZJ4ussC2IKt1m04qeb8aMNFMdmwgWxWMWPf26WhVYddZDSpw13mfhY67rCcTK34lej8uIozPjROR3MsMHRoMHtirDG2uqowXSVO2aMfSX9hTH2FQCqh3sa3BGorfglRO+F3TEmfiSsW1Ug4mcU7hJExh0rHTed8VvEfB9gSPzUeS7NPFT5Y+itnv/PHz+Bb/qZDwGoZ/W0rfLZLTncBeCKH4BKu+fED9FtlT8nO7FHynOPQy874yeIsSMpfppFM6WBrrQdtGxLa/U0UXYvbXZxeXskddGlhBMoC3fJvo5xHJuFu2gWzjycZ75rckUhfnT8pPjlSJ107L6iOJYpfk9c2xezeAT1nAFJGmtNgnB1b4JzUsDTiV5LVMLMg+E0gG2xzPve1Oq5lcTyD72gduDHohW/MIp5p6SbfR5ASjgcm4lradEzfm3HRtux4YfxQoNjDgoi3EVz7LtjH2sdF6dXOPGb1fJIITF1rZ7ZGb9savVBYNoofg0OEO/8wJP4mT//wmEfRiXuhFTP/wHATzPGnmaMPQ3gpwB879KOqsGxghySYUK2qID5mVt8kWg643d+nS/8ymaz6qZ6AulCzwuWo/jle/z0Vk81PbIKRb17V3YmuLw9xs3BtF6dQ0GJOmGomfHjj1e8B0Sl79XhLsnskfTwtGAF1FRPmvGztZYnQVCF4qe8zkFUqvYR7jrRw3Nb41y4CxGlohk/nraaDe8Iorg63MXVhLuE8+8q9gtm/CikQih+lmbGL1fnkNxGc518/7s+gf/t9x5Tjj+vYpdZdHWI4xjDRMElyJUw82Dkhei5dmYTgFs9q8ncrSEnAFGM3Hmrgpjx67gLUfwmwlaeV/zofeRallD8Fmn1pPcTfS4fx6oBeo/rwl32kjn0zV4LtsXmsHry8+CFkbHiOvZDUcMCpO/dunUo84A2KEdLrjhp0AAA3vvoNfzRZ64e9mFUIoqPb6pn+UokQRzHnwLwKsbYWvL3PcbYDwD49DIPrsHxAH3hm74JyOr51M0hABhbPf/b11zERs/NlLmroIVpHJvZTgG+QBpMObFYFPFr2fJCUp3x01s9dba4Mji2pbV60s7yE1f3axe4VxE/xlIF88I6J35llQ7/4cNPY+yHeNVdG6WPTeQmiCLYli0eL33sMDcv1nby9QdAutClGT81on0ahEbn+a7NLn77k+M0/dJJu/QAWfFTrZ4sQ2DHmjkyHdqOnSmLBvjrMU+iJ5ASv46q+IkCd/53y+Ll6vI1RQSBNlGIOKjXSRzHeGZriJVO9lqncyYT7bI0Vh2mQZ44b/Za+ML1gfF9FGHs5YOHui0bUcwft8yiLM8YmoT3yMgqfvNb6HSWVbouB0nfpG0zRHGi2C7S6knET/pcq/NaHAWIcBfNRtru2Mda14FlMZxaac1h9Uzf214YoWNVv0ZTP8psLAmr5wGS64mwejbEr8Hy4QVR7Y20w8CdMOMHgBO+OI6pv+8Hl3A8DY4h6Avf9E2w2edETxA/gx4/gCctfsOXXiy9jXwMpu/JDil+4fyLbIKrWMdkpAskveJXy+pZUjj8+NX9Wopf0cwgYTAN0W85Qh3ptmxs9txCxe+ZW0P8xB8+jjc9eBp/6xXnKx8byBIKIpotx8oUuIsZvwLliBYoq0mqp/qcTBW/S5s9RDHw7NYo87ikXu0VWD0d5XUkZa1fMePHiawa7jL/ruJKMpPZVcJdBtNsuEt67Ok5oOdBxyCsggpx2B75mPhRLmkznY+cPdwlrRFJX2deCbMAxc/PW3Dp75OKOb9b0uPXDXih63a17cAP47mJGB2rWuAOSOfZsgrP3zzwko2UotCq4wC65nXhLnuTQLhSTq20Zy5x35OIn+mc31ixydN7V0dQlwU61mEz49fgAOAFUaZS6Kjitu/xK8DxfMYNFg76wjclfkT0nr5VT/EzgfxGND0eCtVY5IyfTN7yPX7JAiln6yNFa74wFlK8nri2n6R6mit+JbwPw2kggl0IRV1+URTjn/zGp+FaFv73b3pF5TwdHaP8fIhorrYdDKeBWOyIOgdb3wc3EIqfC8e2ckEIU0NL76UTXNF88ga/TokgEYkhJUUlfpbFRA0CkCprVSqINtwljBem+KlWT0qulDcGVLtvWiCfqILJaVSvO7oG1FJ1bYG75rxd3h7hzz53XXv8tODMKn4uhl44t1o29gJ0FUJOhKkq2VNW/OqqIbLiB8yf0mhi9ZTDXRZa4K4qfjV3628OpvjHv/4p4wqNZYCuU93nKc34AcDp1fbcVk/APNlz4od6xe8Qwl2aGb8GBwE/jCo33Y4Cgsg8R+KoYZ4VxfGb4G6wFNRV/FqOhZW2g6eSBbXpjJ8Jsoqf6YyflaR6xpkQinlQGu4ievwUq2ftOge9Qkc7y49f3a+lGKlKlYqBF+Tm1Iq6/N79iefxV09t4X/92y/D+fVu7ucqRGiIovj12zZ6bR6vr7N66pQjOYTGsZjW6llW3k64K6kP+OKNQeZxiQCKGT+1wN1imfAMofi1q6yemnCXKJp7xk8Nd6HjVwvc6djlRaWa6skYSzYIsq8p2X13x36GVHihJtVTc95+6YNP43t/+eNa5YtCflba2Rk/gCcDz4ORl6/ZIIJch/jVnX9Ke/z4Z1+ZSvbc1gjf9DMfLFU4xx7/fW2qp6hzWFK4S7KRUvS5VoWPPb2FX//4ZTx+da/6xktCKBQ/vYOCvqNOr8xO/GSrp4kqGsdxrgOVNm0Oco5ShLvMEGLUoEFdeOHxsHretoofY2yfMban+W8fwIUDOsYGRxy0wKjjd97su6IHbX2Bip89o+I39kN4QYj2ghU/12Y5Ildk9axL/HThLnEci53lz1/bhx9GxlZPi/HZtKIv9+E0yNkVLxYofh/4/A2cX+/gW153yeixdYmSRDT7LVL89FZP9Xhl4ufa+R4/0xCf8+sd2BYTlmQ14ETM+CkkUp2VJFJQVefQdvLhLouwetI5U3v8hqLAXbJ62tljp42AzPtKU1BP10AcIzOnKJJqlY0QtS5hd+zDC6KcYsiPMxsqBAAnEuI3b5ffSDPjR+epSoG6NZhD8Quzil8ZWfr4M9t4+NkdPH51v/A2Y63il1p6bYtXgixP8bNntnrStWTSnbgs0OaQuvE18UNMgwhr3VTxuzWczmTN3ZvUU/z4ZxuUOoc0vGxZePLGIHN89OcohlHNSYMG88ALomNxnfFUz9uQ+MVxvBrH8Zrmv9U4jo2CYRrc/iAiU2eBSgEvtsWwWpF2WAdyd1+dVE9R4L4wxY+J+1ZRtEDyaoa7uHbe6jkNInhhhLtP9DD0QtwceLUUP6B4Uai3enawPw0y8ysA8NgLe3jZ+TXj/rmiGb+VtiMKtUX4jVSrEMd5e9YgORaR6ql5jUxm/Bzbwvn1jlCzBJlP/r8/CdCyrZzdw1bm5Gj+qzLcxbW1KvC8Vs/X3LOJv/WK83jZhTX+PJL7oy/XrD06G4YTKPZafhuWq3OQyb9MxtIqjPT3+21HkGYCqXpXd/Pq8UBD/Mgevj1nsudYU7NBf69afGyPPKkao95CZSoUv8TqWUKWaJZQDf6RkYa7pNdKR0oPFpUdrPw9PguE4lcQWlUFusYO00pIhE/dSKPPNZn4+WFcei6KsCf9jomiMdGouK0lWz1HXoC3/eQH8O6Hn0+PQzrWJuClwbLhJWM3iwygWgZuW8WvQQMTkMWnluKXEL/1rjtXObUKy2JiDsn0btuuzXv8wsi4PL0KtFjXqTzpLIze6tlyzMmSuoCjxcXr7t0U/2ZqFbQ0qpuMwTTM2O0ACBvnC5Ldc+KH+OKNIV56fs3ocQH9jB8pjP22g6EX5Arui2xPKVGwtameXhCVVoLIuLTZBXGctPYgKXCfBtr7Uc/LsIbiNw2yCmawgJLYE/0WfvrbXyPsaowxaV4xWw7v2iwTaZ9aPbN2UHWBLNt9ZTKm2nMBHmgyUGxjNIemI37pjF+6ACbFb3s4p9XTD3LnpSNm/MoXubeGnpgDnWXGL2uPLF7IbyW1EbvjYpKrC3dxbUsK5cn2MC5W8QvnmvE7CoofXc++ovjR56kc7gJgpoCX/YziV/0ajcU5Td87y57xG0wCeEGEW9Lzk4l8E/DSYNmg7/mjHhJ1J/T4NWhQCBHuUoPAUYn7xgLn+wi0SDU9HrnoeNE9fjqVp2ixV9fq6dgstwAgO9Fr75GIX03FT1fODZDil5/xA7KKzxeuDxBGMV5yftXocYG0x08ORRlMQ63VsyVZPYE88dufBiJl0NX0+FGBuwlozk9+XLLOegVx/45C/MaGM34d10YcZxP7/AUUuOuQzivmqyhkUkeF1pnZWYvl+s6e3xmL97Ss+NE1Ls/O9tsO4ji70Ce15+qejvhRqqdk9Uw2jlTF75c//DR+8YNP5Z9wAXR1DkLxq5zxm4rrYzQD8WvbKVkqU/y2DBQ/XbiL/He6ZhnjG2NLUfySxypL5IvjfIIpvT+Hh9gTR58tobKhsZvY5tcSS+7p1YT4zTDntzdJZwVNVFHdOV32jB+9X2WVTybyTcBLg2XDE9fg0d5kaBS/Bnc0RLhLDWWCwhkWOd9HsJXd7SrQjure2EfLIPTDBLSw1iU5Flk9axM/neKXWJMubHRxMSFldeocgGLFT0f86DHkLr/HXuAhDbMofnLoxGDqYyUJdxlOw1zCpFgEKcRuMAmEfdjR2GG9ZCbJBJck4kd2RVmBU4NdAE6MMsql6PGrVvyA7KIwCOOFqdAyWgX2bLXOIVTqHOjPajjIlZ0xXn5xHQAyxepEYmWiTV1/ak8jAFzTET9NuMsGET9lxu83Pn4Z//Xhy7n7KAIVuMswtXpuDTzclSh+w9pWT16B0CqY95VBs4SlVk8vr/gBEGTMVay6iwx3qZPq+eefu4FX/dh7M+deKH6HqCbR56hf4KBYl6yewIzEb+yL36+n+KXnlN6H3pLqHGixLV/78rE2Vs8Gy0Qcx+L7/KhXOgRhdHvO+DVoYAJB/GopfnzhtgzFT5RN11T89ibB4qyeTtmMn34WRqgjNQrcVRsjLVTWOi5eco4rbnXqHID8rjdh6AU5q+fp1TYci2UUv8de2EfHtXDvyb7R42YeW+6/kxS/kc7qaet3vwfTQJAL18qroqYF7gDEwp4/bjpbSFCDXYA8ISc1qLrAnd+vvNvOC+2XR/zU14ETZenxNTN+lvL8pkGI6/tTPHSBE78tyX6p28yga2hfWkTSgvIFrdUzP+NHycBqGMzNgZeJzS9DFMWc+KmpngZ1DhM/xNALcX69C9titZWQdC7OFn8vAil+ZQmmunAXIJ35U0OvFjk/k5/xK34un7u2j/1pgB2JxNK1dLiKn77HT53xI6vnzRmtnqeT3zexw+oUP8YYWpqKmkVhqlFbslbPhvg1WB6CKBajFYdZ72KCRvFrcEejPUuqJ1k9e2bl7XVQX/Hjxx9GsVHohwncUsVPvzNOiw/TY9DVL5DVc73r4AFB/MzvD9BHvQdJxLKa6mlbDOfWOwrx28OD59ZqXQ/aVM+EwPXaNoZSnYNrZUmLutAcTFKC6mhK7k0L3AHgrhNc8WMMuXkpIK+w8OfCu/Bohm3ohWjZViWhT5Xg9AtvET1+OqgJpQR1fi8omPGTid+1Xb4IfvHpPtqOlZvxs1j2vUjnZqghfjrFbzDlr59KUjd6bk7x2xp6uaChItCOcq7Hz8DqSWTsRL+FfhI+VAdeGCmKX/VjmYW7ZK9H2piQz58ulbUIw2lQOevIrdO2+B4oey5EXkPFzgwc8oxfQbjLrqL4rbYdMJYNajFBGMXYnwY4JRS/6udK51SdI245lkjLXTS0il8T7tLggCBvaBz1Socgimu53I4SGuLXYG50HP0isgzC6rmUGT9S/Mxur0tNmxdlM36MMbScfIl1faunlVPnqCtqVVb8ahTCA/r5n7IuOrnLL45jPHZ1Dy+rMd/HHzshfhSyEPJkr5UWV/y8IMLYC+FYTITQFM1H7U9T4ucqChZgXuAO8HAXuh8KQbEzxE8T7sJoVpL/fewF6FXM9wH62c8gipayq5gmlCpWz1ydQ37GT00tJZvvxY0uTvRbmRk/T5NKSsodJXvGcZzO+BUofrrX7+RKWyReAjyMZeyH2Bv7RorWSFhw9XUOZUQkQ/zazmzhLoYzfiapnrRIUjc0iAi6JYptGf7nX/0kfvjXP1V6GwpLMlH86Dn4mgChQ031rAh3oQJ3y2JYaTkZtdoEdK0Lxc+AuBHhUlVc12ZLq3Og45LJ3qQJd2lwQJA/B4+61bNR/Brc0aCdXjXWvgxyqueiIayeNWf8AHPSVYWWIH76uS5dWffz22MwBiOSACThLrmFCoURuHjgbF2rJ/+/Tg2gRZlq9QT4gv/JmwP4YYSrexPsjPxa833yMaqLwH5S5wDwRaNaBA4UzPh1UuKXL3A3n/E7u9rhapP0uGS3AvSKHxFtIpxDzRyZDkLxk62eYWw8o1kHItxF2ayxLSsz50TW29yMmHQbUnsvbHSx2WtlZ/yCOEey6RoisjQNInHNaYmfl++PBPJl2jQLF8VmtsFxAfGzLb4xUzbjR2TsZEL8ZilwN5nx88NIkKWqcJeua+cSkknxk4m7o6njKMLnr+/j+l6xrTGK+EyOTGLLbIyUTKpTlQ+3xy8Jd4nyil/XtTPX8GrHySR0moBU6FOr/Htv1hk/gD7TDkfxa6yeDZaJDPE76uEucZPq2eAOxjw9fhtLCHdRO6uq0NGkps2LMqsnwBf58pd/GMV498OX8ZX3nxK7y1UoCndxbYaOa+H+Myv4qgdP49V3bxbcQxb0IaZTS3RzVoS3v+I8bg48/NYnnsfjL/CS6brET7V6yiXsRBS2R16GgLRs/XzUQFL8uHVxtgJ3gG8eXNzs5mY/idzpCCQ9F+LkIy9Az6CrMlX8pHCXaLnhLupGh6skdhJpVhW/UKP4nVvvaBS/MKeiq8SP/n9xo4v9aZBbXA6n+dlSgM+XyrNWsvpn0rM2Kgnd6bdsoZ7rQBULZPUc1FRCyOpZNeO3bficxl6oVZ87QvGTrJ6WmdUzjmNc3Z2Uqkv0M674mVs9dXOkh0kqRLhLGGdqRnbHPta62etjpZPvoawCEb90xm+2VE+An0vqfF009DN+kdigbayeDZYJeU1UtxbmIBEls4iN4tfgjgURvzqK390ne7jvdB+vvLSx8OMh37Xp8XSV7qtFgBbrRUpP27EyH2zvf+IGruxO8Pdef7fxY1C4i7xQ2Rv7WOu4SVebhV/6rtfj9S86YXZ/JamegxLF72teegYvv7iGn/qzL+CR53cBAA+eq2f1JJswKRFpd5sjSNPOqEDxKwl3UQNwSKGoM8t5abObD0FJXqsyqyctbkdeiH5FsAtQEO6y5DoH1QZsWyzzeul6/GyWV/xOrbTRcW0+dyeFkPhBfkZRTfWk/993mocBqZUOPOQn//qdXuVWTyL2RMYAGAW8jES/Yv6+L2328Nz2OPfvBFIXT/bbXPGruSCmSpGqGT8is6dW2pXhLrogKWHDVxRbEyvs3jjANClTLnseAJIZVgbGytUsQfzCvKp8qIpflL/mAf4aqK6U1Y6L/Wm9GT9SCGdJ9VQ3D9uOtUSrpy7VkwcgdV174eQ8jGL84H/5JD6TfG80uLORnfE7uoqfbgTiOKEhfg3mBlk96+x+rLQd/OkPfVWmb25RoIX3oSp+TvGMH8B3yOXF3n/+yLM4tdLC17z0rPFjpL176b/tTQKRQFcXlrBb5hcVMhFTwRjDD3z1A3jm1gj//i+ewqXNrrFqmT42/z8tCOUS9n6l1TP7BTGYpLUTrmKHpQVTnfP81158Cq+4mN2gcEusnmpJ9mia74rToSjcZd4Cdx2KFL+iGT+VOKiK38WNDgCugKnhLuocoZrqSYvi+8+sAMjbPQeaGhGAL6LjOJ23uzmop/iR1VN3bu491cfTN4eFv7s98mBbDGtdB72WU7vOQU3CLCJX9NzuO9XH3qR4dnHsh0Ldk9ERn83FxL0IRMDLbIV03O3EZtrWzC7LoPMiby4R6TrMVE85ml0+tt2xnyN+K+0ZrJ7J8z7ZNyd+tAGkJge79jLDXTQ9fkkYVr/tLPwcbY88vPsTz+N9T9xY6P02OJ6QNzSq6nQOE2FD/Brc6RB1DkfkTZDO+JndXt4pX1SqZ9rjVzTjl1o9r+9N8CePX8c3v/ZSLUJCi3F5YbY/8cV8W12kc3b5n8lETIevTlS/3XH9+T7+2NlgGXmmkKx4u2M/QyJ0dQ6k6MlphnGc3i+95nXO8//4VS/GO7/zdZl/S4lf/n7ovMgx9boZNRW6cIwginJzeIsAHX/OwmpZGWss/dkpsXpe2RnjQtLnuNlrYXfsi9/Thbu0HQuOVIFA/y8ifsNp8YwfAFxP5vxki6lJsmdRuAsA3Huyh8vbo1JCttlrgTGGlXZ9JYSSZatm/Ejxu+90H3GMwlCRaYHi19VsypmGu1DCapm6RJsU7eQctx271MZI85/yNSZvkBwGoihGFKevlUz89iZ+bhNrdSarJ6Utu9r5bh1I8ei0FDu2k6+oyT6Wj1/4i6dmKnmncyerLRM/RNuxsdKub2muAh3jjlLL0uDORHbG7+haPQNNv+1xQkP8GsyNWXr8lgkiEaY9fssIdxEzfhpiACCzM/7rH7+MMIrxrV9mbvME8oEoQGr1nAXpjrdO8Su2egKp6gfUn+/TPfZAmikksllk9ZQXzaTutRSLGy2UZIViHtD96nr8VMVv7IVGM34dEYevhrscoOKXK3DP72zKBe5xHOPKzkQifi7iWEpvTII/ZDDGeBJmshgmFeG+UwnxU6yeIy8sVPwA4EYy53drIFs9DYifX0b8+ohi4PL2SPu7twYeTibJxL1Zwl1Cs+67reQ5kQ12t8DuWWj1dPNWT/n8lUEofiXzZOn7iYhfseLnh5FQRuXPLHpvHpbiR58Z9P6TSalO8VvtOILImSJNW3ZyNv8iTPwQjOWTpl273Or5gSdu4h2/+yje/fDlWscIAJOCGb+Omyh+C7Z6psSvnnW2we2JOuEuQRjV/txdFBrFr8EdD2EnOiKdJrV7/FpLsHrSjF9ZqqcfIo5j/NrHnsMb7zuBF50yLzwHUoIrz8twq+dsip+oIdCsKWhRpluAE776pWfw49/4cnz7G+oRWCBPYgcaxW/sZ4NCdDY5T5o5AtLzEAjFL6tQzIpSqyfLPiZX/Mxn/GTFxNcoZotASozNrJ5qOAjdZmfkY+yHKfFLyBDZPf0wn+oJ8PNK6oE8/7TedbVWzxWN0nyGiF+i+N0aelhtp+qwDld2xsIuOU6uaZ0qf+8p3t/4zC098dsaejiRPNeVOeoc6DotJH5DD4wBd59IiF/B8+LhLhripwl3MVb8dk0Uv+z7jVvY9beXj103U3dYM370+FR2L8+47o39nHV+teNiUHPGj2ZOVzsO2q5tNuPn6ZNaW7ZVqubRgvnfvf9J49oOAhFSNdWz7dgz1ZZUga6tnZq9iA1uT8ifNVV1Dj///ifx9p/8wLIPSYu03/ZorHnroiF+DeaGY7FcSfNhggio8YyfpNosqsfv3pN9fP+b78ebX3pG+3P68v/kczt45tYIf+e1d9V+DLU2AJhT8dPcH0EocCWWRcYYvuON9+DsWqf+YyvzNUON4gegYMZPQ/yUbskgp/jNd55Lw11mnfFL7muSsXoupyuIiG9LY/X0NVbPXI9fsjhOO/zSGT8AIuBlOA207ylOlug2ofi3NHS+dQAAIABJREFUc2udjOJHHX+6DYdTKwrxG3i4+2SPF2xrFJntoYev+td/jvd8+goAyeqpIUz3nORE66mCOb+toYcTK4ni17Ix8aNai2ya8aNOz6KF/K3EUkqv685Yb4kb+5Ge+GnqHExn/K7tk+JnMuMnWT0LFmyyqiOrav4hp3rS49NrRZ9/UVK6niN+bQcTP6pVqbA38dFv2XCS2gsjq2egJ/Mtp7zOgUjlkzeH+ONHrxofIz0moCp+IdquhZWlKn6N1bNBPavn56/t43JJANcykSp+x5NCHc+jbnCkwIf67SNj9azb4+faTPyOuyDFz7IYfuhrHxSLUxVkifrdT7+Alm3hax8yD3UhCFKjzqTMGO5C509b4D4NYDE90VkEiMRGOeJnZxb92TqHvOJHizh1ho3+XVUoZkWZ4ifP+MVxXGPGj3r8pDqHJfX40fGriaGF4S7qjF9iFZQ7/IC0pmVr6CGOYzx+dV/M7slY6TiC8Mnn+ux6R8yWAWnHn474dVs2VtqOqHTYGno4vdrGatvRWj2v7vFqgidvcDI3Kgl3OdlvYbXt4JlbeuJ3a5haPcn+XMeqSFZPgJPwIiJAyiLV3hQpfhNfv7lA/5YJdzFU/K7u8td1alDnQNUqZTbGXYm0Zu3E/PbTIMpVrxwE6DFTqyc/tv1pgDgG1jr5OgcAteb8+Ow1P4dVATiEsRdp7bu6blIZXnItnVpp4Wf//IuZ1OcqTKVwF/q9TLiLIfGb+CH+7Z98vtKuJxS/xurZAPWsnreGHoIoPpzPjEbxa9CA7/geFcWPCIzp4TDGROz5ohS/KrQdCxM/xO99+gX89QdOz6TSqfULXhBh4kfC7jbr/emJH5+zUm1Hi4JqjxxMQ7g231CQFRlHp/jprJ6KlZF28Rel+BGh1M0KWtJzmQYRohjoFYTiyNDNe/lRtNweP2WjQ1fnYFssc95l4kCBKieTDQ5h9Rx6uLw9xu7Yx8svrucev9920lRPSU0+v9bBC5LVs2q29PRqW1L8pjjRb2Gt62qJH3XiURjMyAtgW0wb9MMYwz2nenhKY/WkUnVS4ciKXCechFs9E7Lklit+J/otMWdWSvx0PX5OdgMEMCd+1/fTVM8i8kBEwWTGL6v45TcXgHTu8iBBj6+Gu9A1pLN6AqiV7Lk3Ti34PADHYMYvUdpUuHZ5uAuRqe/56/fhU5d38eEnbxkfp2yvo/PIiV+9cJf3PnoN/+aPn8AHv3Cz9HZC8ZvR6vm9v/wx/Ohvf2am321w9JCxelZ8FtDn/sRgE2XRCDX9tscJDfFrsBC0nSNE/GrO+AHpznjLOZjn0HZsPHVziKt7E3zdq87PdB/C6pl8WFKAwMyKXwnxk0vRlwH1seXSbrJHAdDO+MkLTap2SK2e9Bqpit+84S6J4qchDXJCaZmdUIX6fEJREru8GT9XeY84Vl7xU99HtmXl5rLo+Z0gxW/kiU7HV2iI36qkHvDUThuWxXB2vYObg6lY2JYlbwI82fPG/hRxHOPW0MOpFT4nqCNIW4md7EZCaEZeiJ5mhopw78m+VvGj+UVS/MiKXEvxC1LFr2xmaytRFon4FSkjY19vC6TPNdWqaxTukhDwONZ3ewLS+01O9TSxemoK3IHDSfaka43mIenzlJ6HqrrR55JJcixBTgdVq3yKMPH0gT0txy6d8aOf/b3X342T/Rb+0189a3ycMiGlupOJzwlov2Wu+D38zDYAZDZxyo51d+TXUiYJH3lqC49d3a/9ew2OJur0+FHi8WH0/YlUzyOSa1EXDfFrsBB0XPvIED96M5qmegKpzW5eQmD8eG5aB1Cnu08Gvd6k0NBc08zhLiUF7kVzVouCaltVH0/u5RO/Y1uwWFbxS4ldYt21KbAhu5hblOJX1eNHCyWTVM+0By3MHPMyvlxSK6xq9bSURXmUs7PYTEosVUqmuy0bHdfCzsjHI8/vwrEYHjy3mnv8fttOUz2lc31urYM4TndzByaK32CKoRdiGkRc8eu42kU5zR2S4jf2ymcv7z3Zx+XtcU5dIZWT1E2y8ZouiuM4zlo9S8I+yOrZcW20HaswrXRcQBLo+iwK5ylCEEa4OZgKa3eRwpRT/ErCXWRVJ1BUZcJhJHsGYsYvG+4yVdwDBLJ+1gk62Z8EombH2OpZQOZdm1UG7jDG3zMvPrOCm/vTwtuqkBU/+jMpfv22g7EfGqnFHxfEr3wGiz67vTCq3du2O/axPfKNEnwJQRjhH73rYTxyeTmF8Yc1p3q7QA5nK5vxi6JYfA4fBvFrUj0bNADwT9/2EnzXV9x72IcBIB24nU3xOzirJwC8+SVnZiZUtJgLVWvSnHUOOjWgqER7UbDFXFxa5yAv9nuadEKAny9PExSh1hXIdlhguTN+MvGjxYzJjB+QnZFKEzWXZ/VUSaVa56ALl7EtS9xm5PHZT9kueaLXwtbQw2ee38WD51a1r9FK2xWLpH3pXJ9f5yExpBTIIT86kNVza5CqcOtdV6QoyhBWzz2yeoaFSiLAS9zDKM4FCNBjCasnKX6GahVdr+0KxS+MYmyP0lnCIiUzSizF2nAXTY+fSbjLzYGHKAYubfJ00yKFKZ3xk6yehTN+esVPJpWHofgJqycpftLMIZD/TqAZv1pWT2n2Wu5wLcOkoKKjZZeHu1BiLGMMXdeutTDWKX5TP0TbsYxnWUdegEdf2ANgoPhJz2O75pzfs4kNu8552B75+L1Pv4C/esrc/mqKa3sTfOk73ouPPLW18Pu+U0CfM2tdt/S63Rn74jPsMPr+mhm/Bg0AvP0V5/Hae04c9mEASN+MdcJmaGd7GYtsHUhh/NuvvDDzfaSKH1k9SfGbjfgJi6ImOGBYEKm/KKjzikNPUfwS4qTOpKmLZk+xcqo9fqLAfYmpnulziSTFz+y14+oP/8JLy9MX/zHddgoUP8vKnP8wyofLOBYTITycPGVnPzcS4vfI87tamycAPi/kBSK1kxbTlAhLAS8DA+K3PwlEuujJlRbWuo7e6pkQv5uDKaLEhqurciDce5KTnqeVZM+/fGoLFgMeOLuaPJd6ip+6+VBk/dseeYjjlGBu9Fyt1ZOUGZ16qavaMVH86PW/a5OH9hQpTKniR+EuxVbP3VFRuMshK37U4ydSPbObROoMKM341al02Bv7WcXPgIzxpNb8e5+nehafv6lkI+64Vi0lTT53tKCeBhG3ehpe5596blfMBqvVLCrkz+66yZ7PbPH3ZR3LbeqmqG8rrcL1vSn8MMazW/oKmAbVmAri55Ret3Jn6+EqfseTQh3Po27QoARk8awTREI7qwel+N19ooeza2286UF93YMJXCk9Eki/AFc7sylz9Bmmt3qGxqrVLCASS4RiMM2WdhNxUpW6lrJ7roa75FM9E6unpni9DhwDxS+K68348eNKFRP1uSwSrrDCKoqfzUShNcBft/yMHxOLZZ1d8kS/hUee38XOSB/sAnDVJI45ceQzfonVM1H8rgrFL6160OF0EirzuatcYTjZb5dYPfnCMohibI08jP2gVPGjSoenlTm/P3jkBbz+RSdEYi+Fu5iSFvW8tgoKudXgnCLFjxboOnWoKxS/rNUzqpinokqNu06UK35TneJXYvWk4Ck13IU+qg+jkJmORSh+YTnxo2vRVGmK4xj7k0Ca8Suf0SNMC62e5T1+XhiJY+aKn7kiklH8kp7Z1OpJynb58374WW7z/GsvPmk84wfwOb86oI7NwTQQ3xtVoGtzGUmQ9N0yPqRS8dsB9D292nFLA5BuSMTPZF520UiJ34E/9EJwTA+7QYNiCMWvxtXdETvWB/OW+LY33I0P/tM3G/W7FSGdi+MfkPNaPen+dIvCobfkcBc11XPiZxRGEfSikJC20oFGyh4RGrXHjxZB89ZStES4SzHxC8K40qqooiPNe+3NGdZThlbRjJ+iBoVRlAuA4cSB/1lnl9zst8SMXpHiR6/HYBpgf5Kqu5s9F23HErNBRKYKw12SEvfPXeMBD5SAOfLCnB1OtpJd35tWWj1PrbSw0nYyit/nr+3j89cHePsr0kCmVPGrZ/VMZ/z09shbg2yIzHrX1aYf0s647pqmf1MVv6KwFkKq+HHiV6SQkHJlNOM38nEqOV9ZO3EkPrNMX8NFgq4Teg5q56c6971a0+o59kMEUSxZPeed8dNvFBDI6glwMltX8aPNgqkfpg4J2epZcY4+/sw2vuTMCh48u4oXdseloS3y86ib7EnvyzhOk4GrQO+zOh2MpqDrhTb7GtSHsHp2nNICd/psBA7X6tkofg0aHBHQvFidGT9dCMKyMW8/m5pYOS9ROMxwF1tRL1WFUcz4qVZPZcZPnctxFMWPbCE6wlYHjl1s9ZRn/MiqWEYwZMjhLqTurC+D+DlkAcwTPz+MxWItiGJxbgiy4jfShIpsJp1zRcEuQEqWBtMAQy8NvmCM4cJGF1eUGb+ycBcAeOwFTvy41ZM/vhr6sD30RDDH9f1JYSAKgTGGe0728LRU6fAHn7kKxoC3PnRO/Bup0aZqlWr1rFL8qCh+vdvSBlnQLJbxjJ9k1S3Ctb0JbIsJBbZooZyf8bMLbYw7Yx+nkuciKy5BGItAqkNR/JQ6Bz8qD3fpuDZatmVM/GjeNBvuYpDqWTjjx8orNoJIWG87ro1JDSIy8SPRGTmWiF/HtY2snlEU4+Fnt/HaezZxfqOLiR+VdvTJm3bbta2e6fvSNOCFXndvCVZPeq2GDfGbGV4YwrYY+i2n1MJ584hYPZsZvwYNjgjozVgn1bNzwFbPRcBRgkv2xjxooz+jipgSlvwib6hYLxeN3IyfmuqZkMCc1dO2RGExkA/OSMNdVMVvTuJnlRS4SyE5lCB5JpldqwLvd0wizoWCu/jXnRTRVo7UkerL/x6EcW7GkBMH/uexn98QoBL3B87qg10AifhNguTaSm93YaMjiuFNwl0A4Ilr++i6Nnotp7Dzbmvo4SXn1gDwZM8qxQ/gAS+y1fP3H3kBr7tnU8wiAnKqp6HipxCKol63rSG/dk5UhLvQwqfU6imnerLqOoeru1OcWW2L81do9fTNrZ67Iw8n+zrFLxbn7DAUP2H1TJ4rff5RVYXOBbLScUR9ThVEzU5HCncxUCm44qef8Ytjfe0OwAvcheLn1lf86FxM/EiyxluZzZoiPHlziJ2Rj9fcs5kLatIeq6z41bZ6Dmurr43V82iD1OqOW57qKSt+ddNgFwFaTzSpng0aHBHM1OPnHmyB+yKgKnT7SXLcrCXraYF79t+9IIIXRksNd5FVsjiOc9ZSUlXUmbSWavUMyOqp7/GjL4l5Lb3U96gLiZHPy9XdCVbajrFNVg7HWK7iR4qoWueQDcOhkIbMbRTFTyVPRFSKbJ5ANhBlMMmSx/PrXbywQ+EufBFbtCFzot8CY/w4TiZqEqlHe8picHvk4YFzKwB4XcTICytrNu492cPl7TGmQYgnbwzw+NX9jM0T4Oe741rGM36qkqSq1gTqqSIivd51MZgGOfVNrdSQQdenbNe1LJaZsdPh+v4EZ9c64v1WZC30Qr5Qs5L7bzs2gijWLqx3x744R+qMH5Giw1H8sptBvjLjp7v2VjuOcZ2DOntdZoclxHGMiR9pyTx9thWek0y4Cz8fptbGnOLnpxtpQvErOUfU3/daifhd3SuudJDVb92mRhHGXohre1PxGWMa8EKPtwyr57Sxes4NP4zRciyuVBsrfgdv9WwUvwYNjhjmUfxUK+FRBi3KaJG1J3VFzYIixa/unNoskEvPx36IKIY+1bOizkGdn8r1+CXR5NacH9hlip94HcMY1/YmOLvWNr5feVFIFrFlEL/iVM+s5TaI8j1+ljQHqLNLUr/dyy8VEz86t1sjD14YidAPALiw0cW1/Qn8MEqU3+INB9e2RGm8PAsHZO1fEz/EyAtxfr2LtY6D63sTjL2gMnTnVZc2EEYx/uZPfgD/5x88DgB428vP5W630jYvt87N+BWkPG4NPax3XXGOaEGu2trqKn6OSbjL7gTn1jriGMsUP5kYEdFUSUkUxdgd+4Koy3UOYcQJTsu2DsUmpyp+ariLjvittJ3aVk95xs8Lo1K7rbBYasi8+EwLChQ/qSOSnpOpHW4ahNjotsTvpCnIabjLoESV/fgz29joubjvVB/n13ki7JWdEsUviGAxvoFTJ9WTkjMF8TO2epKauzyr57ghfjNjGkRwbU78ypS8m4O05uZwCtybHr9CMMbexhj7HGPsC4yxH9H8vM0Y+y/Jz/+KMXZv8u9vYYx9nDH2SPL/Ny/zOBvcXrBnIH4i1fMYK357Y3/mYBf5/tTN0KpI/UWAPj8DaS5OVhh7RcSvsM4hO+NHr9GkIDChLsSMX1m4SxTj6t5EzEmZQE71FFbPJSp++VTPrH04CGNt1x8RP53i98DZFbQcC19+38nCx6cNimtJp558bV1Y5yXuV3cnGHqBOPdFILsnpV/Se0BWEMhGttlr4cxaB9f2phj51VbPt7zsLH7uO14Dx2J476PX8Oq7N8SCVkavVYP40WLarlb8aHEDoNDCWjbj12s5eMXF9cyspUzci0AbFi1l4yT3XMIwo57Tn1Ur4/40QBTz5+BKPZAAv8Zsi6HXtjE6hALsXLhLRY8fkCh+hsRvoMypUqJwWUBL2SwybU6WKn4UPpVc36Z2OFnxm/ihOI5suEvx8/7087t41aUNMMZwerVdWenghXyhX1RVUgSyX79cKH6manu2KmeRaBS/+eEFPJG2KPCKcGs4xaWkauZQZvxCUvyOz3pRxtJWcowxG8BPA3gLgMsAPsoY+504jh+VbvYPAWzHcXw/Y+xbAfwEgP8OwE0AXxfH8RXG2MsB/BGAi8s61ga3F1Krp/nvbPZbYtf5uEDMr0nhLoshfori55UHbCwCjLGkXywScz4Zxa/E6jkapV/6tKimxRF9MNPiblLQjVUXrm3BYvreRzkd9druBG98cTEBUtFxs1bPXsteSuAQ3WeR4kcLoyCKc8llFktTIXVdeC85t4bH3vG20t1QOreUHtlXFD+AzwYNp9VpsqdX23j86r6wmIpwF8n+JYJS+i7OrLbx3PYIcYzSHj+AX5dve/l5fO3LzuFPHr+OF53qaW/Xa9nGalWuzqGg9Hxr4InnBADryYJcTT9MUz31mxDv+f6vzP5bRYH72AuxNwlwdn0GxS8hKqqVkaL6N3otODbLhrtEMVzbQr/lHI7iF+kVPznRUsVK28XlbbO+NlWRlclx0SZUqX23wuo5DSL0+07mMSdeNdHh1Q2huM4mSrhL17VhsWLiF8cxnrk1xBvv432+tsVwdrVdPuOX2FLXu/WI3zMJ8SPFz3TecpmpnqTajw6BiNwuILW649jwwkg7agBwq+crL27gU5d3jRJyF43jrvgtbyUHvB7AF+I4fhIAGGO/CuAbAMjE7xsA/Mvkz78B4KcYYyyO409It/ksgC5jrB3H8RQNGlSAFt517Hx//4334E0PnpnbAniQSJUl/sG3Pwlwz0n9wtTo/pRKBcJBWD2BNGZel+RIj62zemZ6/JSUwdQOmyh+wWIUv7e87CwsxrTzlHSIfhjh+v4U5wyDXYBsuMve2F+KzROQFFF1fk9JVw2jOFfnIBe4jz19F17VFyKdW1IDVjTE78rOOBf8ogP16Z1cyVo9ZWWMEgM3ei2cWW3j05d3AZinrVoWw1tedrb0+dRW/Byp9FyzEN0eebj7RPp+LlL8JiUkQQfHKg93ITJ+drWTs0rnnovUGQdIpEZJrdwZJ69/14WdJMcSgjDiil/LPtxUT+rxo3AXxT0gY63GjN9EIeakLPLXSP/+LqudcZP5Yr9gwZupc3DNFT8/jBHFwErLgW2xJNUzVfwY42mLRc/75sDDyAtxj3TNnlvviGoWHaaJwrPZa+HJm4PS4/vEs9s4s9bBxY0unrk1wkbPFaoP2WmrQN8Vyyhwp++eJtxldvjJtUvvxWkQah0ftwYezq13wNghp3pqNn6PA5Ypb1wE8Jz098vIq3biNnEcBwB2Aajb498M4OGG9DUwhVD8alg9+22nMHr+qMK1FMVvXqunsugnUCebbDtbBhyLIQxjoc6sSs9F1DlUzfjlrJ5qqmc4d5UDAHzZvSfwI3/zJdqfkUJ2fW+KIIprWj2zit/SiF/FjB9F2vvJolwGEfQ4jo3skjq0HQuOxURReJb48dfryu4YA4MaEWH1TK7PtmOhZVuZxSARvxN9bvWkxes8PZoyem1ztUqd8aOAIjWef38SZN4DgviN9FbPsmoKGVbyPisCnZNzkuJXtKteNOOn3p7UnPUen1mUP2OCKIaTEL/DSfXUh7tMk0WobnNntWM+46eSuCJVVEbZOa0k45lwF/5/E+JHnzuk7vFUz6zq2S/Z4Hh2i6tw95zsi387v94ttXr6STiQidXz+971CXzPf/wYoijGM7dGuOdkH45tod+yjcNd6DkuR/Ezt3q+4z2P4mNPby38GI47vDCC6zB0kutNF9wy8gKMvBCnVtroOOUhMMtCk+q5RDDGHgK3f35vwc+/hzH2McbYx27cuHGwB9fgyGKWcJfjCNvOKn57ykKxLtRgDwIN55MSsywQoXj42W0wBjx0cU38TIS7KLartjrjF0ZwbSaUW1fp8RsvyOpZBnodn08qCc7WVPxosbU7J5EvQxHxI9JKxCCM8jN+djIjNg2ixC5ZnzwxxrDScbRWz17LwUbPTRQ/A6snKX5JVQBjDGtdN7MY3JYSMs+spmE7s5BWHVZqzKfl6xz01r2hlw222ShS/JL7MyV+VXUOtPFycqUlzfgVB4m0pY0UQWqUBRvZU0nxy4a78Gus13IOR/FLnhttCMkF7kVpsiuJ4ldWTk5QrbhFqqgMKq9ulxA/L4xwczDFr/zlM5mfT4PZwl1SWyeP08+mevL76beLyfkzSd/l3ZLr5Px6B1dKStyF1bPnYmfsl76eW0MPn72yh9/+1PN4+tZQKIurHbd2uItK/P7Dh57GE9f2je6j6r6rwl38MMIvfPAp/NFnr871eLcj0jqHYqWaqhxOrbQqax+WhSbVsxjPA7hL+vul5N+0t2GMOQDWAdxK/n4JwG8C+AdxHH9R9wBxHP+7OI5fF8fx606fPr3gw29wXCHCXY7pm9IUrpVaM4MwwmAaiCj7WWAVWD1f2B0ndpzlkBACpQ1+5KktvOz8Wob0UJ2D2junq3OQyUyqikqK3wKsnmWg6+7yNid+tayecqrnJFhKsAsAPHh2Fd/3pvvxlV9yKvPvgihH6YyfOsBOVkHa2a5KxixCv+VorZ5AWunAw2PMFD8qOgd4pYNMkLaGNGPmitsDiyN+s4S7qMRPVYBG0+xzX6sIdzGtKLFtlgtwkrEn1YgIklGk+AWhMuOnJzWimqTnwlXqJPyQz5GWkYplgq71bisbbKQG18hY7bgigbgKEz+EYzHxWrZL1AzxOyWKnzx3+Z/+8ln889/6jHBlAGSfTArca4S7pEEutojTF1bPZLNspV1s9Xzm1giMQdgvAa4aT/yosKqBiN9mrwUviAqP0w/Tn/3rP/wcruyMcW9CMNe65uqrp7F6xnGMf/mez+LXPvpc0a8ZgV6rKsVvNM3W9TRIQddDp2TD4kZS5XBqpV1Z+7AsHPcZv2USv48C+BLG2IsYYy0A3wrgd5Tb/A6A70z+/HcA/GkcxzFjbAPA7wH4kTiOP7jEY2xwG2IWq+dxhJjxC9MkzHkUItq9UmPGr+xOcH69M3M/oClsi2HshXj42W28/kUnMj+rU+cgL0TVVM/pARA/eh2phLyO1bPj2AgTIr/MGT/HtvDDb30wd/+2ovoW1TnEcTr7WUXMirDacQTZWVFqSC5udPD8Drd6VvVHvvLSOu4+0cNLz6UK8Xo3qwJsjzysdhy4toUzq+n56LqLmVtdkayen3puB3/6+LXC206VOdS2kydX1J3Zl4ipa/NkRdUSN6lZUcLDXYpJx55UOE7vpTJboX7GTw134bv0610Xjp1N9QwjrtIfuuJH4S5E/MoUv7Z5cfjYz1aekIpXZvWcBFmVUIaswj76Ap9VlRe/XpAS1jTcxVzxa7tpj5rc4wdUWT1HuLDezSjAclCTDvR5TWp2kd2TXue3PnQWV3YniGLg7sRSutZx57J6cts6sF2zQF4FvX+rruFB8vO6hfV3AqZhhJZjC1eOjtSR4ndypVVZ+7AsRA3x0yOZ2fs+8ETOxwD8WhzHn2WMvYMx9vXJzf49gJOMsS8A+EEAVPnwfQDuB/AvGGOfTP47s6xjbXB7YZZUz+MIOXqf5pkW0eOXU/x2xtoI+0XDthg++dwOJn6ENyjE7+xaB47FMot2QF/nIIcxqGXHi0r1LIMtWT1ti4nwERPQzvokiJY641cER5kbpaj97G3432kxNuucnGzvXGnlFT+yelbN+N13egXv/ydvyhDstU6e+FFC5hmpV7EqOMYUfD6NW/9+5N2P4Mfe82jhbXWpnkCWCJCKpz739a6bV/z8sNY5IKtuEfbGgQhbEQXuZUEimRm/NJRBxs6IJ9S2HRuOxbR1Dv22eTLqIpFP9UzrHIqIH33OmhC/iR9lLJsmVs+xV2zflWf8Pntlj99eJn5htsBd/XnxcaaKXzrjlyWg/VLFb5gJIwLSTa+igBf6vKYKiSIyRO/ltz50Dm96kLu7UsWvBvHTpHrSn+v0CGrvm6yeFa81EWc1nbcBhbswSfHLf+7clBQ/OQztIHHcFb+lxvTFcfz7AH5f+bd/If15AuBbNL/34wB+fJnH1uD2hXOnWD2lAnf6Yq2jLqlgjMFi+Rm/F3Yn+PIalQSzwrEsfP46T3b7snuzxO/cegcf+pE3Z2x6QN7qSb1Q6X0uJ9WzDKQ07459nFvr1PpyoN3y0TSY27o7C8TrlShCYRK1L4PeVwOh+M32eq5o6joIFza6optrljTZta4rSp4BPh+0mRQ7g1BtAAAgAElEQVS9L2PGr992EEQxPnV5F4+9sFcahJSmembDPuTrmCpU1NdlTUf8vNB4vg/gi5WyGj9eC+OAMZbaCkuqA0x6/HbGvlB1dHUOPNzFOZQePzoWIme+VOBeVO+TEr/qxfvED4WNFChWRWWkc4H5x6dzcnMwFXbyrOKXT/U0WRznZvy8MB/u0rLFtani2a0Rvual2eTb84L4FSh+iTV/PSmNp/RXFUSwVzsufvTrHsLZtS+KDr+1joMvXJ891ZPed9sLIn5+GMNXvodkEPEznUu8kyDqHGgDSav4ceJ3ot/K1B8dJNIZv+OpLhzPo27QoAQUUHHbh7tICh0FiVycM4DFsaxM8EOQVBJcOCDFDwDuP7MiyrhlnFnL201bDreNRZI9S16IpnbYxaZ6lsGW5hDP1iTidOw0x3Dgip9ijQ00PUqp4scXLrMqfmmhtSXUawIle8q3q4N1ZcZPVvxW2o5YEFf1+JmCLJm//GEetFEW9a8mz7Y0ChDZxVQb7XrXyS0YxzXty2q4ioq9sS/mCWlGtlzx06lZ+VRPuk/bsrJ1DlEk0hlHfpizmpvi/3v0Gr7jnX9lFLgig6511+YbX3KdQ7vAHUAhWiaVDmMv+5lTFIAjQ+3+k0Ebfp96ble6fTKTG0aIYuTCXWaa8Qtk4pcqfro5zME0wP/P3ptHzZbd1WH7zjV+8/D6DerXc0stdQuERkyQkCwEwZDEeBEWdjCYwYATnBBM7BWTGK8kTswyJAZCWAkYsxZDEi8jYMlLxkaBBYhBoLGFhlarh/de9xu/qaY7549zf+eee+65t25VfVM9nf1Pv/6++qpuVd1bdfbZ+7f3nUFQCHYBgJ1+q7bE3c8W+utddWIt4Yjbj21c3erin/zVp/k53285zXv8FFbPgCt+ixExkaTUzfnR66etnmXI4S4TBam7M2C2/Tx9Vs/4zQpN/DTuO9DC9H6f8RPrHGjnd9HkTdkGdnvgI05SPLA2v5LYFPS+yfN9dZAVCdl6ZhgGHMvg9QTjYDZb3DwQZ+IurDS3eQK51fPW0RkRP9nqqZrxM4pWz3ln/IjQqezJ4nk8l+KXWT2JBOwNQ24nMwyD2z3nDaaRQcf4m5+4AYARn6hqLi5mYR+knKpm/GhxKCt+HdfGKCySjbkUv7pwl0nEZ4VNM7t+hOfy+ZtH+NMsir6k+FVYPQ/GAX/9Has4Y8gVP89GmqoXe03wsZf38fvP3alUJ6tAz80xzcL8YRBXK36zzPhNouJnTrHHr+JviITVzPh97OW90u1lha7lVs9KyZBn/MZBDD+MYRg52awKd6FC9Qc3uoWfW6aBnb7Hk6Fl0EbdWqb4Vc3ZkZVTlVq90rZxOGmWsKpK9aRNiHsLKn7ieVeX7EmK6XGGu/yrP7uG//EDf3Fs93dWyMNdshqSQG31pPGJs0v1ZI+pUz01NM4JvlRSPU3TgGGwD6Hre2Ns972FbYyWlLjHqxxOUfGT5/vqQIsg+kKXrZ4AIzNc8avZxT8uiErzLImeQB4pf/vwbIifJVs943KdA1f8FrR6EllSEbsC8Zvj/lfbDiIhdXFvFGCjk9svye55XJsA9ByCKMFbMpty1byavDmhmvEbVih+rOS8eL+jYHrJvYhpdQ5M8csf17HMwkL5J//d5/G3/sWfIoiS0hxcneJHi3uqbQFYSEKasmuU3ud5kz2JSNVZKFWI4hSmkZFc4fNP7igUMYvVcxwUFdkmVs96xY/9/aey+T4gV/RK86OWCdOYXjEA5IpVK5vx86OEfV7aeZdh17MRREkp7OelrMrhQUnxA5hN/9VD9YxfSOEuNONXYfU8nFTPsK9kCatN+vNUqZ70s4NxWDv7Og2iglsX8EJWz7GQmroofv1j1/H+j8mh+csHssjS92BVuMtWluCsUz3ngyZ+Gvcd8lTPMz6QU4BjmgiTFNf2R4UY7XlhZZUKBJodPA3Fj943eb6vDnJQQhiXF2u2ZSCMmR00iJITt3qKu4AzWz254scI90nVOVQhnxsVrZ5S11+28ORWzzk3GyjJs6tQDHf7HuhlnHfGD2CLuUkYYxTEWO+KxK8F02hegTANdIxrHQff8MwDAFCZflgKRFHM+FHku/zadFyL/47fNoxnsqyamapfpZCwGb/8vJPnaAd+hMNJhD947g78KG4043cwzhVXxzT5+UVVCtTjB0xPRawCEak6C6UKUZJyq7Ft5ZtElDCoAilPzRS/RCJ+DQrcwximoLSJEOscrmywz3xa/JLqRLcxDKNx8mFR8TO54iceOymdA+l5v3iv3OFH2Op5PIlRhmjt82yz0upJ9mbV5yH9rEnACz3HqKD4sX+n6WJzd0Wrdp3VM3/tjkv1e+ne6EyUr+NGqc5BafX0eWcrWZJPG9RzqxU/DY1zgjzVczkvyllA1szre+OF5/vo/sT5n1cyxe+0Uj2vbLRnsqvKi2ZVIINjmYiSRAgvOOEZP5H49Wed8WPHdlZWz3KdQ1r6ciML9WCyaLgL+zu5ygFgC3Aqvp+H+NHrdjiOeGjDhkD8rm51sN33jq2ihNSqv/L0RaxlymIl8ZMshKoZP674qayeEjEa+dFMllVe21IhbhyOowLxcywTgaCQEMn4rU+8UkliZSWD1XKw99G2jEJ4EACe6gksoPiFlMY5299HccI7UUVbuDwvLGImq2cQoy24DLjVs4aMTcIEbcdSnp+io+FNr1nP7iv//ANQOL+azkFNeKBMNjuVzfiJr8FmprRQsiLhxbsjrHccZZ0QqzqZvgmy3nGn1jmo5n1nSVilcyNQKH7AYgEvovpdR7RFJ0AV0Z0FUcwcP2ehfB03aOazLpTo7jDAVp8UP53qOQ808dO470ALm5PunTsPsC0DQZTgxv4El9fLu62zghHJ/P9vHIzRdS2sLFAT0RTf+MxFfM9/8MhMf+NK81Gq7i07s2/lC5uT/dgzDIN/IcyaskqLrFtnZPWkRSUPd4mTaqvnwjN+TvZf9d/TBsA84S60AD0Yh7g3ZIu59U7+Wn7/Ox/Fv/q+d8x8v1V4dKeHN19dx7e/40FOaKusnlX2SOWMn0LxkxeVoyAuEcQ6yORexuGkaPWUK1Mm2b//7adfLZSFA4w4GUY5DMYXLNaWaXCrXSjsnC+q+E3mtXoKAUaWafDd/EAqpxdhmQa6rtUo3EVOEm6a6lllQxZJ3Zc/uM5vL96neNwzK362MOMnvb9Up3PrqEj8Xro35L16MlQqNUGsnljrOJXE62gSoe/ZyoU2XetN1DpVnYM4m7dIl18QJfwzpqnidxyVDjf2J4iSFH6UzB2MdB6QpsyR41lmXmsknbdRnGBvFHDFz7PPxuoZZ58Zy7rGPN2scA2NU8CXkuJnmwZeORgjiBNcOgarp20Wgxde2Z/gwimUtwPA9371bKQPUIS7xOX6ATajlNaWIh83LMNAjJSrVk1BiyxK9VTtoJ8k5BTUWKH4mZz4hTCM+Yk0KTxVih5Fwc/TtUcpgS/cGfLrYl2Y8et69lxKYhXWOi7+n7/NiCTZ2qoUP5n4qWb8RpWKn4UwTgsbHOMwnkl1pRlUFfEL4wSjIC5ZPcWF8iSI0W/ZnPiLipBhGHAts/BcojhBnKRwLXaMjmXyx85j0QXFb84uP674zagAiNH7tmly+6mfLUKr0GvZjWf8RDu0PJeswiSMC4RLhGPn1+OXZ4pfHu6SJ3MS2m6zxTG9bi3HymLyE4yDopV3NwtFIis64cW7I34sMqoCYQByaLBjXW07lUTocBJWdtQubPUUFb/hYorfesfFzUMf49oZv+NV/F68Nywcw0mHl50U8nRdM5srLRO/e6MAaYpzMeO3zOtLrfhp3HcgheJ+T/UEmCXuxWyw/jhm/EyjWK78ysF44aTQkwQtonLFLy7Zs8haRpaQeWfSZsG8il9LmPFjsy+n+xHtSHUOoWLGj4jgwI8q7WhNQAu5XgWxI+uyagZwGl57YQVP7Pbxs7/7Bdw+ynufTgNEKOsXu/WBKLQ4lC2cNMsnhnUM/Wgm1ZXeP1XAC5E5cZbKyVwFhEkU46se2+IuANla7dpF4kebMkXFL1uA8xk/M1f85uzyy8NdZrV65gFGjpWHu6jcAyL6LaeZ4ifNyRmGAc82p6Z6TlP8Oq6FJy70s9sXrZ7iZ2A7U++aHCf9LR3v4SQshGHtZBtZNw9zxY85TsbKYBd2nHZlym0QJZzIrnWc2jqHqnlnOg8Px82tnoVwl4Litwjxi7lDo86ufNyKH33/A83SW88rxGAiukbk50NOmO1MeW45bFN3kVCeeRAr0q6XCZr4adx3oB3t+z3VE2CLuBeyKO3Lx0DQbMso2EVuHEy48nIeUbJ6KsJdHIuFSdDi5zTIlG0a6Hn2zDZFPuN36GOl7Zy6lYRInjiDVaX4HU6ihQrQidBVvUZf/cQ23vXE9lwBN6Zp4L967+N4/s4QP/8HXwSAQrjLSYKIX124i6dQ/AJJ8VP1G9LrTZUOcWbxmknxI+IXlxdLeYiGYPWUFL9xpgh+7VMXAKCUkuvZVpH4SXNnrM6BrMSC4pedD3MrflFS+G9TsDlWIdwlyT9L6sJ/ep7dbMYvTEouA882p/T4JZWfU1aW5vzaB1bgWCZcy+RuBjnVE2g+B+VHCSzTgGOZfCZxfxQW1MOeZ6PjWnwBDgDX98dIUuA1G2riV6XkpmnKXmNLmPGrSvUcR5WKXx60M51Eid8TFG4kntuLdOsFmeIHsMClKgyCiCtWxxHuQlUawPxVKOcB8rnL1LzieUtKMynPPATmlAmvVvw0NM4Z6Eu8xqVz38C2DP7heBxWTzFqPYgS3Bn4pxLsMi9UM35yEp6dKQz0pajqxjpumKbBv5xmQd7xlWC1ffpOfJtbPVnqY5xU1zkMJtFCtiKe6llB/N7xyBZ+4TveMvcX7Htft4tnLq/iE9dY0fXaKc1LTrMsVgeiFOscVK8LJ37ZfdPs1izET1T8hn6EDz77Kv/dIS/KlsNdBMUvU7C+/ukHsuOXiV9xJlBMi2SPn3flcaunZXJb68KpnjMTv/wzwxbmD6crfjavGahCnDByI5M4z7Fqj7Oum9EwDPRcG2+4tJrdl8k3teRUT6D5jB+zl+aLboARE/nYd/pewerJO/wqZvzoPJbfV/lYVzsO9kahMm32yA+VHX5A7hyY9l4AxXMjFr7nCIuGu5DFvM7qOfKjbHwCOFiwOxAoKn5NlN3zCvl8UIUS0YYDKc8tu3lP5XEi0cRPQ+N8gff4fSlYPTOSu9F15w7ZEGEZ+W78zcMJ0hS4eApVDvOCz0fFeT+TSvEL4yQPdznhOgeALSBntXkCxUX0aVc5ALlNOkpSvjgvpXryHr8QHWf+c46UvnnCW5rAMAz80HufAMDsYLJ6dlLoTVP84unhLiNfPbfXkayeZIuctc4BYAvf3/z4DXzvL/0ZXs7i+MkuJ557pXCXTMH6qke38CPvexJf8+Ru4f5lG6Os+NmC1ZP+W1D8Fu3xm3ERGMX5Is4W1Eh5FlPGSsvBYIrKVNXHN9XqKQXCyPgX3/lm/Odf8ygAZPN49NzVVs+mBe70d7ShczAOS7OGO/1WIdyFzp0qq2eVAk4Em17j7Z6HIEqUlk2WNKs+x6kKolG4i3Aec4IvKN+LhLuwzbpM8asNd4nR82ystJxjUfxeujfi1TfLXOlAnxM0b8uqGorPhyzG271iD6t8u5OGKu16maCJn8Z9B/tLivix53gcVQ5AXg8BAK8cnF6Vw7woz/jlYQEENuOXCuEFJ/+x13Ksud4TcZF12omegFjgngpR+1KPn3E8it/uSgvPXF7FG6+szX0f0/BVj23h7Q9v4tIxJN42BZt7nNLjJ9Y58LCPYp2Daraxw0vO2X3TAnOWkntLCHe5m4VZvHrIrnWV4ufaueInKli2ZeL73vkItvtFZVvu/fMlpV0kV2KdQ8thgQ5zK37hfIpfKdwlC6MRA2lUaGL1FCsSRHjSHKQMufRdxpse3MAmLX4FS5xK8WsrkmCrjpUekz6HBn5UUnR3Vjw+NwsAL++N4domX4zL6PJztngM8oYAbZTRuSjiaFKt+AFso6JRuEsYg5YF9FpRuItnm3OHu1CAUce14NpmrfI2DFi1yVqnOsymKdI0xUv3Rri6xdTWpbZ6xsVNC+WM39EEG123YAcFTl/xi5dc8dOpnhr3HSwKd1niC7MpSMU4jmAXdn8i8WPl7edZ8ZPVEjEsgOCYkuJ3ClbPn/m2L8fOPFZPYZF1FsTPyUheHCdCypqk+Fl5ncMiM34tx8L7/85fmvvvm8AwDPzcf/amU7VAGQZTr2rDXYT32TSNUoDKKIiVaaZtPuMX89sB81s99zOrGVmolDN+Vj7jRySuLiBJJjW+tMC3TJOrLeI5Rq9bE8Xvlz78An77L27hX37nW0qPM9eMnxTuwkNSajaJ+q3yexzGCV66N8Ij2z0AuRW3rPhZtTN+flSeC6xCS7R6VvT4NTn/VYofO1bZ6tnC7xze4v//0t0Rrqy3K2fqqxS/fKaLPRYlIL96OOGhNQAjN0eTqHBOymhiuwXYc+y5No78iIfNEOHYXWnNbfUUSUvHtabWOXRcm6WYLpjqeXvgYxTEePJCH8/fHmKyzFZP6dxVJXbePPSxI2w00QbFWcz42eby6mbLe+QaGhXQit/8sIRUzxunWN4+L8QZPzksgGBni7lJg0XrceGZK2tzvW6mafAvvjNR/ESrZ7aYkTdQSDGKsh3u845+y+EzIaeFrlfsLovihCsSzOqpIAKFVE/1jF9XtnoG81s9kyTlC8+bNYqfI1g984Ck6ve9rPgVSRQLd6FY/aKqzBbN0xfwH792gD9/ca/ws7lTPYVFHJtxTpQESkavZWMUxIVEwd/8+A287yd/jxNqUuJa0nXiOfVWz7FU+l6HVla2DlSFuzQvcKf3tVWwIktWzxUPoyDmpPflvRGuVAS7AKgM7ZGP9QJPDC0qfuMwRpSk9Ypfy5lq9YyyzSyaLc67JIn4eXMTMdFi23Hqid/Aj9H1GPFb1OpJ832P72bprsus+JXCXcqK3+2jSeGzvMX7/k7X6rnsip8mfhr3HWhhuswXZlPQTvVxKX6WaSBJc8VvpXW8fWfHDbHHT54ZIdiWiTBJMQ7ynqrzDNphP+0OPyBX/Opm/MT/n4VwfCmh69kYCATmpz70HL7mx/8/HE7CktUTYIRnICgWo6Bqxu8YrJ7ZQ0dJyq1mNLN1OI5gmUbhsVmqJzsXaJamzi7NSGx5xs8T7JQRV/yozoGdU13PbpTqOQ7ikn2RK34zLgKjOA93cSwzK8Nm910349dVFM7fOvIRxinuDYn40VxxOQCntsdvyoyfCJHY+RXhLk1TPVWKnyrcBQBuZQTt5XsjXKmxUleF9gRx8TUmh8TNgyLxIzttVaonQFbP+g0DUuXo+4wIH52fO/35FT9fUC+ZtbYm3CWI0POsYyV+T0q1HssIej8KM37S87l56GNXUPzoGpl1rndR6Bk/DY1zhjzc5YwP5BTAFb9jmmESF2U39ifnWu0DijN+qvkWAHBMA1HB6nm+P/ZIGTnTGb9sZgVAKRRF3FCRe+Y0GJhlMV/8vXh3hDuDAL/4By8oQ0PWO25h0Vk140cL8rFk9Zxl1pLUtThJeW/aLUHxW2nZhRqRuRS/uFrxsy2Dl6TLmwsd12rU4zcMIsRJWojin3fGrxDuYjJ3AD/mGuLH3wuBqNKxkxrGrZ6y4mfPn+opQ1wg0wK4UODuWAhidY+eiEkY8zlM8f2VU5B3sg61W0c+DkYhDidRZZUDkIcdybbY3AJs8GNe7zilGT9uP65V/GwcTSFRdA73ZOKX/XdnxcPeKFCmik4Dn2O1TXQ9u1LxS5I029RhM36LEr+X7g5hGsCjO5m1+H6welakeiZJitsDvzBCwWf8TlnpjJNkqYWF870C0tCYA46ddzLd7yCL0nEpfqaJQqrnPMmUpwnR6klD+s4Uq+f5V/zY8Z0F8bOFcJewyupZUPzO92t5VpCtnkTq/s/f/yKGisCM9W5x3mfox1wpESHXOZCKMkuiLzkikjTlvWm54lcuyhaJXJM5WbmjLrdNZuEuZrnHj86ppt14cp1FmqZzWz3DRAh3yeYZqzaRRKhqO+i4SL2tDXepUCmSrJuxae1MSwjBkAMyAKDtZnY4iWgmSYqf/tBznPwXFD+R+CnCXQB2zry8xxSnKxvV3z/8nK0KdxHuf3elVSiHB/KahjrFr9+arvj5JeKXWT0j9t+dfgthnM7VIynOhLZrrJ40m9vzbKy1XRyM1fUVTfHivREurrXR89g1u8xWT1mtFi3MAHB3GCBOUj4Lym7DbktuntOCuFm0jLj/V8YaX3J4+tIq/v7XPYm3PrRx1ody4iCL1HF0+AGMSMbZF9HdgV9K7DtvEK2eVYs1ZvVM+K543S7+eQApI2dR52CaBkyDfbHFFVbPguKniZ8SPa8Y/LE3CnFhpYWDcYhxGE9X/Hy14teyrSz5sqj4zWX1jFUzflFJWXGtvH6hqp6gcPuS4kepnmKdA+uJJKsnEa+NrtvIbkeEl44nSlLQqF0wh+JH57iTJQAHTRQ/pxxaQqThiBS/QP16eY5VeZxEUGZT/OrDXcRjITx/Z4B/+sHP4gOfeoU/Lk/1FFwR8muwS4rf4YRXOVyus3q6asVP3hAAWLKnPONHxex1n4crbXtqqidtRpQVvxiWaWAzK1WfJ9lTDDDquNVhOnSudDKrZ5yklSFQTfDi3REe3Ozw93iZrZ7lcBezQOioO1IMd2mdUbiLqt92mXC+V0AaGnPAtkx871c/cu6VneOAbRpYadnHNg9mZgXuaZriziDgX4bnFXkUfnUgg0P2rTDOIuPP9wc2KX51KXYnCSrYjppYPTXxU4LNquULuoNRgDc/tIF3PbENoHyOrgnEL4oT+FGiVPFM02CKAp/xo3CX2a2eTPGTZ/zC0nknWj15WMlcil/RiZGkudWTzqn1xsSv2FunShFtCpbqmSeOxkkqzGxNV/zEWUN6P0i1rJqJrJvxG89oSS/UOUQJbNMoJGx6jnpxTH9DRMsXCtyLil/xvV5p23Bts6D4vaaiww9g7y1Twep7/ABGKktWz+y1rOrxY79zEERJLQGgDYg83IW6JFO4lon1DvuumyfgRaws6bh2ZUARkbyeZ2O148z9eISX7o3wmo0uJ+qnTYCOE6G0ccuSb/PnI5e3A2dn9YyStFRztExY3iPX0NDAVs/DkxdWju3+bNNAkqQ48iMEcYKt7vlW/AzD4AXTVYs12zL5jN8ybAbQ4ussrJ4AVXok3IZXp/jpcBc1OtKM394oxHrHwQ++53H2e8nGuZFZPdM05XYwVZ0Du29LUecwg9UzuzwGfoQgStD3bByMQ0zCOJvxK1s9F1P8ijN+dP6EcYI4O8coVGij42JvFCIRkjJ/7Dc/je/4hT8pPAbZBokkiQvEmVM9xXCXrFw+V/yqn2fHLSt+udWTLeYnFTORdQXuTV5jES3HLKR6yp9/7QriR+8RJ35Rwp9vccaveH+GYWCn72WK3xirbWfqxmPXszGQrZ5SuAsA7K62cGfgF+YRjxRJszJIDaxT/aqsnkHE3v/1jIjNE/Aipnq2axQ/Om+7WZ0DgLnn/A4nIe4NAzy42YFnsw7MZSZ+yjoH4RpRKn5nmOq5zOEu+ltbQ2OJ8Y++6Sn+BXYcsDLF7+6Afflt9c+34gfk8fF8x1BW/CyDpXqGMbeGnGecNfGzMise2fDqZvy04qdGz7N4H12cpDichFjruHjjlTX8wne8GU9fWi3cfr3jIso2XGhxWEXmxIXlOGAqzSzzJrRTTdf44xf6+LMX93D7yMfhuGz1dCyW6plk1xDQINVTWIDKCzoiWXFSPsc2um7h9QKAZ28c4Pr+uPAYstWzoPjNmuop1DnYUo9fneLXUYW7BMVwl6q54roev6pAmCq0hJ4+cU6PwK2ecgpq9vivZimak8wRAbD33M6+C1SfmTt9D7eOfOyNwtr5PkLXKyt+KofGhZUW0pT101Gw2OGYZvzqw10AprTu9NW3IaJNGypiuItrm/x8m4v4yT1+FQRsIFg9adZ2XuJ3I7smLq+3YRgGWnaz2o7zClW4S5iNHFimwWc/txWpnsf9vD/47Kt4+d4I3/VVDyt/H+lwFw0NjbNCR9g5PA5YBlN77g7Yh+zmOVf8AFIY4srFmsMVv+TcJ3oCuTXrrIifYzG7G834yQXutiZ+U9H1bIxD1vHGAhzAFYV3PbGDzV7xuiKb2d4w4BbRKsWvK1jJhkE083tAC857Q3aNP77LEgFvHk6Y4idZPel6YnOys6d65sEXFO6SVYbEop04J37s2PLF952BXyr/JhJDhGdSUPxmI35hnPBz2rZM1uNHalRNQJiqn47I/vQZv2qrJz2XOrWxeF8sITRNU7XipyCoQK74vZotqEXFD4By3o+w029xq2ddoidBTrmlxwPkcBd2XbwqVDocTULYplH72U2bFXUkKlf82G251TOrV6Fzb64ZvzB/Lu2aAne6bnuezYnmvFZPshPT9wTrvVviGb9SnUPRvnrraIL1jlM4R0npPO46h1/68Iv4+d//YuXv4yTln6PLiPO/CtLQ0Dg1WBZL3LtDxO+cz/gB4FZP+YuDYJtMsVgWq2fLNmEaUIZ7nAaoxDqUyrXz35eDIzSKIDvZMIh4mfdap5rIr3fJZhY2UvzEcJdZbJ4AS+4FgDuZ4vdYJpFc3x9jFMSKcJc8ObdpqicphECutOQzfpnVU2EnXlcSv4ATKjoOuVewOOM3q9UzD2qg4BlxIV+FNk9YzQnNuJTqqZ6J9DJyLFpaCZMZFT+6Bv3sM1A+Zr6AlogmEXKyeoqKn3jMqoCb3RUPNw8muLY3ru3wI3QFBVx+fJFc7/IS9zzZ82gSYaXt1M5mc6tnI+JHil9m9YwTOLaJ1bYDwxrZPOcAACAASURBVGDX4KzI6xwsdBwbQZTX4Yjgit8xWD3pHKPPmpZjlVTdZUK5wL2o5t089AuJngCzHXu2WTq3F8ULd4e174sOd9HQ0LhvQFHrtCjc7i2J4ifUOZQVPza3M5khIv0s4TkWVtpOIaDhNEFdZpWpnoao+OlpARXE+S9aSNIOvwprKsWvYuHfEYjfuKLovQ6kuBG5enyXEb8v3BoAKKcncsUvTgUiU710EJN2AbagM4xcObaFHsG8x4/9bFMifkGU4GAcIogTvrgWlSvR4khYJNzFzsJdVLUIMrq8mFxQ/CSr5ziM4VplKy6pFqIySuCvccP04TzSnrkeZJWSFtCy4kev571hgHEQI0pSSfHLQzZk7Ky02Bx4lOByE8VPCjsCoExhzolfrvgdTsLaKgcg31SpVfwo1bMU7pLw92il5fCNmlkgpsB2FBsCBDpXmOKXhbuMi49363CCf/JvPjO1d5FUZXpt5N47Qpqm+N5f+gg+9NlbszylU4d8zfHzmit+6pTxVsXznhd+FOPG/hjDIK58D8h+uqzQxE9DQ4PDMoozfrQDf55B1jK5B4hgZxHtkzBGewmsnk9e6OOZy2tn9vj0etH8VYn4CTudusdPDd7x5sd8IbleQ/w2hPki3s3nqRe7LDWQLXSGcxA/4gVErq5udeBYBj7PiV851ROYRfHLFSj6r2vlabq2GO5C55hVVPxozkpU/kgxEgkE7+4TrZ4zz/glcIQ6hzCuDooSwas1/LLidyT0+KlIMi1uVcfK7aEzzPgBbJ5QtmsCNeEuAkGmdE7xWNs1ip+4AJ/X6qmy5m92XTiWUUj2PFJUjMhYa09PyOSpnpLVk4W7sGNY7zhzKn5Cj1+FtRbIg4C6noWWY8G1zRJZ/fWPXcfP/u4X8JlXj2ofM1f8nOyxLaXV83AS4YPP3sQfPX93xmd1vPjxD34Wv/OZm5W/p/PBsWTFj/381uGkpPgBOPbZxpfvjXk1TFU3ZLTk4S7nfxWkoaFxarCyVM87Ax9rHadkmzyPcC0WH19V50C7+Mti9fyBdz2KX/zOt5zZ4/M6h7g4f0UoKn7n//U8C3Crp6D4rddZPTnxCznBqVP8xgHNkEUzk28ze//Izr3RdbHd8/AcET9FqifAFsrjMIZtGrWfC3T7vFC9GDhiC+EuoWT1JAJ8NyN8dIxAvmge1Sh+bcea0+ophLskzcJdeLWGSvErEL/y+9Pi9szysVYFwlRB7HDzo3JHJCciNcTvhTtDAEWSl8/4qcNdCFcadMjWWj2FxzRNAzv9Ypff4Xi64rfaiPgl/FgA0eqZwsmOYa3TrE6k6r49yxIUv/J7K1o9AUZYD6Rj/uT1QwDFc1+Fgc/+jhRMNuNXfszbWVVLVdLoaeFffvgF/OuP3qj8fRCxwBRS0jyhoy9JUtw+8gvnHaHlmBgf42zji3eH/N9V1mGt+GloaNw3oIXP3aHPbVfnHZ7DFD+5B4hAFrPBJFqKVM+zBrN65jMq5Rk/TfymoSsQPz7j166+nvotG6bBrJ7TFT+LB4qMgnjmWVDR6ulaJtqOhZ2VFr6YLf5lqyddP35EAUn17zmRh0BU/ITrjkhWKNqJs5+1XQttx+IBG+LidyB1FwLlVM+Vtr1QuIslbBIBbCFfh45r8/eC/R17bLLhTcJYScy54qc41qpAmCqIIRh1dQ5lq2f+2C/dI8XPKv2dSvHbyUrcDQO41ID4sddJ7vFTb9TtrHgF4nc0iaYSP9sy0ctqSapQrnPIw1287Bg2GvZIlu+bevzMWuI3CmK0HYt/hq62nRJZ/dT1AwA5YasCbS50svepSvmiGgSZeJ82JmGCV6R0XhFkuSXQdeNHMe6NAkRJqlb8jtnqSZ+DQLV1WEwCXkYs75FraGgcO0wjn/HbWoL5PiBT/GoK3GlReeRHS5HqedagSo+oasbP1FbPaSAyNvAj7I9CmAZqF6+maWA9Uxuo76xXFe7i2HwRPwrUxKIOtF65Nwyw2mGhGbsrHn+/ZcXPkxS/psSPFtqBrPhl5w8LECpXhmx0XdwbsgUXzRoDasUvD3eJ+bFXEb+7A185dxUJQQ1k+aTXV5VoKUKsKRAVNVJjqipk6H7Vil+x93AaSJEbhzGCuFznUFVyLSp+L94dlR6T/q36zNzJ0jcvrLQapY+y1ylGmuaBJ7m1r/j5cmGlVUr1nGb1BDISNa4mbWQHJuJH81ss3IUdw1rHwd5wdqun+N1D3aaqc23gR4W03rWOUyAXh5OQE4/bUxW/GD3P5rPgbddSFpkTgawqlZ8Ff/z8XWVozTREMQseulFD/ORNC5pxnYRJXt6uUPy8CuL3/o9dx7M3DpSPlaYp/vylPeXv6FoAqomfVvw0NDTuG9imgThNcXfgLw/xy8JdquxZtNAcTKKlsHqeNfI6h+L8FaGo+OlwFxXE4I+9UYC1jjs1rGetw3b/aWasitAR2UjTFKN56hyy47g3CvhsFCk4QP2Mn18xsyZCTAEFGLlREr+KAKH1rsOrJqYpftzqGZLi5yjJ1KsHE/zln/g9/OPf+nTh52maZmXMZPVk/yUVr67OAUDB6knvm20aXI0ZhwlaSsWvOL8kYjKr4idY4lThLvTaTyQFisI0bNPAi5niJ5K4XPErH8dGx4VtGo0SPQGmgMdJWgzhyRJI5bTO3ZUWX+gDbM6qrsOPsNYp2yZF0PMl4hdkVk9RaVrvuHOFu9Acq2ka9YqfH3E3AACstl3sC+Ti2czmCTRQ/PyQPxcgszwqHpPuZ7ig1fO5WwN8y8/9EX73c7OHxNBmxs0jv5I4yom09F19Z+DjJpW3K2f8TOWs7I++/1n80P/98cJmA+GPnr+H/+Rn/hB/8cph6Xcv3B1ym/3hpErxS/SMn4aGxv0ByzQRx0zxW4YqByCLRq+pc6D/b6JWaFCBe1KavyKI/6/rHNSgBdnAj7A/DmurHAjrHRf3hgGGAUuCrJova7sWkpQtNuepc6D3L03zNETqTwPKip9jFRW/ae95rmblip+rmPErqMrC5sJ6x8W9bAF/tzDjl6uchElUtHqutp3SIjCKE/wXv/JR3BsG+PzNQfF3Ulcl/XcURLBNYypZ73pipyI7lp2+h4HPiPkkjJXpnHVWzyYBOiK4JS5koTSyUmgYbBaxXODO/v/iWhsvZXNNTescTNPAg5sdPH6h1+gYu0LKLSEQLJYidrPE0KEfIU5SDPyotBmhwlrHKZAoGXRedCWrpxzuMgzima2Dfpif43R9qGf8itfratspzJGRzXOt4zQgfhGf7wPI6lk+n/IZv8UUPwpaujuYnRiPBTs0WU9l+NKmxeO7fVxeb+N/+sBnOEETP6cIVUrn0I/wmVeP8DufKRNVei6qmdAX7g7x+kurAGoUv1grfhoaGvcJLJMtpg7G4XIpfnGN4icsKjXxmw6q9Jg24+fZ5Zh6DQZ5xm+t3YD4ZfNFoyBCp6K8HchnekZBnBG/+cJdAKY4ALniZwmKBUGsZ2gSkORmc3HijF9R8aM6B7HHr5jsmM/4BZyocqunMKtEKlZu9SzP+P1v//7z+JMX7uGB1Rau7RWtZnmAEdU50GOVQ1JUEKs1iABur7RYF2CWglo346ciGE0CdESIsfcqxQ9gi+MS8csUlgsr+evSVPEDgF/+7rfhR973ZKNjzK+HYh+jo3iNL6xmJe6HE66cNlL82vVqnZ9tQND7WrR6sp9d3eoCwNREzfJ956p2h4fplInW0I94jyBAKn9+zJ+8foAHVlt4fKc/lfgdTaKi4ueqg4244ifN+P3WJ24oFa8qqKzWTSGe51V2zzBOC9dc27Xws3/9TdgbBfhn//ZzAKCuc1DMNgZRwjd1fupDz5VUP7pWVfbn63tjPHOFpWrXzvjpHj8NDY37AVZWdg4sR3k7UC5wl3eoHVPcxdYfedNgW6zHb9qMnw52qQYtmod+hL1hWFvlQFjPrJ5Dvz6whRSDg3GIOEnntnoCueJHM1srLbtkvSvWOSQzKH4x/zuvEO5CdQ7MTmwYxWNaLxA/H1eyugDZ6unaJlc48nCXotXzYy/v459/6Dl885su41vefAU3jyaF34dSZYmVPddRENV2+BE6rsWJKC2Id7PF6dEkyhS/8utFIRWqRXCTAB0RJaun4rjbjoVxUC5w92wTu6stfq0XFb+sx6/iM3N3pdWIkAF5Qq0Y8FJFUnf7eZcfWe1WpoS7AMCqNC8nw49ieJbJ32vR6knK41sf2gQAfPgLs1UfiHOsHT7jpwp3ka2eTGGkc/JT1w/w+kur2O57DWb8iqE3VYrfrYoZv3/465/CL/7hCw2eHYPcUTkLxgXip1b8gigunQ+vv7SK/+E/fgOiJMV6x1FuQrA00+Lzpuf62E4PH31pv/R+8lAoiTC+vDdCkgJP7Pbh2iYOx+rnqmf8NDQ07huIn7ub3SVS/IQZP3mnXCt+s4HVOSR8R7yqzkHP91XDNA10s/TN/WzGbxqYxTHA0K+f2yM1kObfZn0fCsRPmvGTEz0BMdwlxTiMp4aOqGb8XIXiF8UpQkUf1kbHxZEfwY9i3BkEvCeOKw7ZYm2z6/IFJdn4+pniRzv8f/7iHtIU+JH3PYnL6x2kKfCKsPCMJDszhbsMg2aKX1dIq+TELyN1Az9i1ljFe3l5vQ3HMvCF28PS72a1pIt9Z/KcFMFzzJK6QUrsrqCiFArc3Wqr56wQFXBC1bHurubE72gmxY9tnKhmugBwG6xhGLyvEShaPbf7Hh7b6eHDM3besftmr1ddj9/AjwqbOq+/tAIAeP9Hb+BoEuL5O0O8ISN+dxqkepZm/GrqHMQZvzRNcTSJZlLvaIND7mNsAvG1eOVArfhVbVp885su4/vf+Qje89pd5d+pUj3pef2Ntz+Inb6Hn/rQc8rfy+4AqnK4utXFSqt6I0GnempoaNw3EG19W8ui+Nl5nYPYA0QQiaBq3kajiLzAnayexdfTNA0Yhk70nIaux0qr98dhbYcfYb3rIogS3B36lVUOQK600sLwOBS/Xa74lY9TLnCfmuopz/jFVTN+iXLnfCP73Nkfhbgz8PHAaguuZWIQ5FZP02DHmtc5xHAsAx3XRprmHW20SF1tO7i0xmoHRLtnvrlRDHcZ+VEj4td2rTxhNXssei0Hk6hSvbMtE1c3u/jC7UHpd1Wl75XH4EiKn6KCou1Y5XCXTHG7sJoHZoiP+57X7uI7v/KhkgI8DyjsSCQfYRXxy4jzy/fGMyl+ax0HUZJWhpj4Ya48O5bJ33vZYvj2RzbxkRfucWLYBL6gVk2rcxBTPd/1xA6eubKGn/x3n8PHXt4HAE78DjPFuAoDKSim7VhZN2bxuGmmTiRffmaFnIX4DY7N6lmh+MVJKeGV8Pfe9yT+6V97Rvm7OuK32nbw7e+4ij/8wt1CUiwRZPnvXrjDQo6ubnaw2rZre/zMY7guzgp6FaShocEh7r4vzYyfZXHFT2UdcrTiNxNYj18+4+codjZtxSyYRhE9z8a9YYBREDcMd2G3ubY3rixvB1idAwDcyeyQsxJwkWitZkrkeseFYxnKEA2xwH3SINylpPiFFXUOcYooTkvnF5W43xn4uDdktTKsADxfeHZdGy1hbs3P7KR5aAr7+SCIeFDO5axv7vp+HtdeFe5CATvT0PVyxY+Hu2TE5cgPMQmqSdwj2z08X0H8ZglN8oQZPz9SK7KqcJcgU6nEbjRR8Xvz1Q386F95XePjqINS8av4vO55Np65soaf/tBz+DeffAWAWomWQT2ZVXN+YtWFbRp5gbug+AHA2x/exCiI8Ylr+02eGgAUQnUcy4RjGZUF7qJCbxgG/t7XPoEbBxP897/xLABmb6RN17oS94EvK375BgB/zlGCvVEIyzQwzJKA6W8B9RxiFUbHZvWsmPGL0kabLTI8hdWTSG7HtXF1k81tiuqd+Lkh4oW7Q/Q9GxtdlwXv1KV66hk/DQ2N+wHionBpZvwEq6fqi0O0ZGiVajrsrMSad6wpvuDMLClQoxodz8L1bJHTxOpJt7l5OKm1b8qK36wF7pZRtnqapoGdfqtC8ctmoniBe/2ygSxvfMZPVvzI6pmkiJKkdH6td9nr8PztIeIkxVbPzdTTPESFFb3nMe4UriGnZQ6F3rQHVluwTENS/IrhMvT5x2b8pp/fbYfNVcVJylMTudVzEmESVZO4h7e7ePHuqKTQzGr19GwThsEW/GGcNg93yVQqUfFr2h04K1Spnn7F5zUA/MLffDMe3u7hFz/8IoD6DkwCkcMqe54f5vZdcokAxR4/AHjrw7PP+cldlWymskiQ0jTNwl2Kz+UrH93CVz66iS/cHuLCSgvbfY+HmFQFvKRpWp7xEzYACHezWpRLa22kaV4fQqE5syl+81s9iYzurnh45aAi1TNO4Da45mS0bAtBnBRqImgzputaggJbroEpKX53R7i61YVhGFhpV1s9k6TshFkmaOKnoaHBQR9mrm2WvqDOK3iqZ5wok/DEnbkmi7kvdViWgTBJlB1rBK34TUfXtTnxaxLuspERniRFwQ5Wut/SjN/iVk8A+LFvegrf985HSrcvpHrWEBl+e0nxKxW4C1ZP1awMvQ6fv8mSFTd7HnqeXbCadT2bWbyoziFTFen6zolfzNUm2yomWAJCuEt2THQsTVM96b0Yh3GhzgFgVtUwTitJ3CPbPURJipfujQo/n1XxMwwDnm1yW5rquFtOOfiDNsouVCh+x4lKxa/iNd7ouvjl73ornrq4AsNounGSEb+KLj9xDo+snmmaFsJd6LGfvNCfac6PFGcCq/koz1Sya7v8vfrDX8vSUalGYLvH3pMq4jcKYqQplIqfWGdCfYiUVioHtKjmEKsg15bMAiKjD2/16mf8GibZimhJG01A/rzarsU3e8X3Iyd+kuJ3Z4gHN9lM8WoN8dM9fhoaGvcN6MNsu+cdy2zHaYAWlUM/VgYRFGb8dKrnVFCdQ57qqe7x0uEu9eh5Nu+JajTjJ9ym7rVtZ78j4reI1ZPscQDw7tfu4unLa6Xbi0RuHMw+4ycrO0WrZ3kBRcTvc1nnHrN62oLVM0LbsTJVRbB6Olb+2NlCU7bDXV5v49qeYPWUFD+xx69ZnUOW4OizoAzDyC3ylMpYReIe2WEdeM9LAS9Vpe91aDsWDjMVR/UZqJqDIuvjdiHc5WQ+Hzs81VOwIcZJ7eOtd1386ve8Db/63W/DahOrZ3b9VHX5iZULtsWsnlGSIk3LgWBszm9PWY9Qdd9yFcFIer2JbKk2dd54ZQ3/6Bufwt/+6ocB5LUFVcmedF+FHj+F1ZOI49WMzND1kls9Z5/xmy/chX0WPLzdxZ1BoJxdDKIYrj37moO+00USNxKsnl1FyupImA3OHz/Btb0Rt4autBzljF+SpEhSrfhpaGjcJ6CB5WWxeQL5wnRQEcggLiz1jN902KbJ56/Y/6sVP22brUchtr1hgTv/27pUz+wcpiJllYJQhyrFrwq0KPajpKCaVMG1isRPrnOg+6MAIXkBRfbTz91iil9u9RQVP6ug+E3COqtn/vpcWm8XFT8puZaHuwTqTSQZYpDHyGeElOYkadFdtdn08DZbYMoBL35F6XsdWo7F1QnVcbcds6TuUOl4y7H4psNJET/q/JQVv2ldhf2Ww62X05DP+NUofnY+h0eBYEBZJX37w5vwowQfe6nZnJ88x9oRQn8IlIpZZc3+9ndcxVdc3QCQf/9WKX6UdqpS/EQyR8TxwU1J8ZvD6rlQqmd2TI9ss82OVxV2zyqb8jSoCC+pk52C4pcf90Sh+F3fHyNJUVD8DidRKSWWXAJNezbPI5b3yDU0NI4dtADa7C4R8cu+cAeTSJkKZhcUP01WpsE2jSxxkXWsmQri964ndvC2hguyL1WIO/tNrJ6iqlGb6pnd710Kd5nxnC4UuDcgfvz68sNGjyeTL1kNsbjixwrc5WvWtkysth28cIcpYVs9Dz3PyhWHIEbbtVl8fVAsiSeCGVQQv8vrHdw8nPDfx1K4i7jJ0Yz4ZRbGIMIwiNFxbXi2BdcyuSJb9Zmz0nKw3ffwhVtF4ldVAVGHlmPVWj2V4S7C7OXuSiubFTwZFcMwWL2JSDTmtfZVIVf81OEuIjlzM6tnVQXQWx/ahGGgsd1TVi87jl3qzatT/GQ4lomNrltJ/LjiJ6V6AkUyQ1bPvBIlI29UYD4D8Vukx49IGW123FDYPeusv3VoK4kfKX4Wf70Lil9QVvzuZZ+npLauth3ESVp6vpPsM2eZ1xKa+GloaHDQomxzSRI9gXyhc1Sh+BVTPfVH3jSIBe6qRE8A+Gff8kZ885sun/KRLRfEnf0mxM+2TB5bX6f4uRZTT3i4y4yKHy8rNw30G/wt3Z7KjKddQ4ZhwLVY4FKSpAjjtGLGL60sQt7sukhS9tirbYf15WWL1nEQoesyxc8XLFuebfHrnyt+QYyesNC+vN5GkuaKQ8QL3PO0R8IsM36jIGbHlf1/v2ULil/1e/nIdhfP3ylaPatK3+sgKn7KGT9FuIs4e3lhtXXiC1lRtQWqe/zmRcthqa7VM34x3xggq2dQofitdhw8dXGlccALqaeEtkLx44EjDa/X7Z5XTfyUih9ZHkXFb4L1jsNJMZFRUgxHYVzZeyhDVNxV+OPn7+LHP/hZ5e/GQQzLNLjy+Iqi0mHe82Ga1bPjlK2eeZ1D/jf0uhGRJOWeLNTy3y5zuJleBWloaHBQ4t+yVDkAubVs6EfKHeRCqucSf1ifFpjip7bhaTQHLfA822ys4NB8W93i0DAMdBwLR35uZ5oF9J6utp1GCo9hGHBtk0ebN7mGPNuEH8XKhXVe4J4gStR2P0r23Oi6ME2jQBqGPlPEWo5U5+CIVs/clibOS1KlA835hZKdWXQHNFGjRKvnMMhDWXotO5/BrHm9Ht7ulaye42AexS9/f1Q9fi3b4kScwGYv2W0f2+nx/sGTglh9AQDhnApPHdY6TqXVU1TlHMtEGCf8/Ve9129/eBMffWm/tkuPIJJKgJ0XMkEa+jMSv7638IzfrUMfO/1WqVuQjiVOcvI7DSNpPlDGb3z8Bn7qQ88p5yLHWWDRA1mCrKrSoYn1VwWynk8K4S4RDINdF3QtiSmrE0781IEwQO7AkDcS6DNnmcPNNPHT0NDgoEXhspS3A6IVrYnit7wf1qcF22J1DlGcLnVy2VmDduObzNERKL1wmh2MFieGMftclmEYLCmxQWAGwbXy1Mgm1xBVrJDyJi6KRcUvitWbC6SQ0gZULyMNaZpiHLIev7ZjIcpqR0qpnhRbL4W7XFlnljea8+NzrFYx3IWewzQUw11yW2nPs3HraHr4ziPbPeyPQm4zA4CJ0AnXFC27XvGjY5hIYRZEeH7ovU/g177n7TM95qzouhZXbYHjV/wANudXa/V0isSPWz0VoSJvf2QTQZzgz1/cm/q4pToHJfGrn/GTsd2vUfwy8tX38mu4asZvu+/l56nCrknWxWmgvwmiRFluT4RbNb9HFSUtx8Jm18UNxW3mVvxstdWz7Vh808qxjEKwUG71zJ+HrORV1YMQQVzmtYQmfhoaGhy51XMJid8kqqhzEGb8dJ3DVLByYzbjt8wltWcNmsVrYvMkUMjGtMRUIhhd155rLss2jUal2ATXNrk9rMmCx+PEL+Z/T3AKPX7qzYWNLju2rWzepteykaRscTYKInRcq2BtI8VFTBSl3jSRRF9YbcE0BMWvos6BPYfpz7MQ7hLE/P97ni28XtXLLDngJU5SBFEyszOhJZRYq8NdSPXIF79i2XvLsbjKelKQrZ7+Mc/4AUylqQ93oToHZvXk4S4KlfTNVzdgmUajOT9fIusdhbU2t3o2e2+J+KmsmINM3e0pevzEOofbR4z4kXWcyKdI/EYNS9wLM3J+WdXbGzHCfX2vrOZNghhtlx3fA2utUqVDmrLz3psr3KVscaV5W4KYAAyIVs/yz+jzjTpN5RJ3Kr1f5nAzTfw0NDQ4OPHrLpHVU+gZUy16CoENesZvKqysziFMUlgVM34a0zGP4keL72mqAC3k5118mIYx03E5lsEXQE3mZF3bhB/liop4XRbCXZKksDFD2Mg+f7Yk6+vRJMIkTLIC9zzMQmX1nITl3jSHuvwyq1mcKX5ERu15Fb8gwsjPiV9fYcFT4dEs5ZACXogoz6omiOdBVbgLUFSDToJ41aHj2sU6h6i+zmEerHaqu9fEkKGS4qfY4Oq3HLz+0urUOT92HqcF8thxy+Eu3OrZVPHrefCjhFu6RaiCYlqS5TFNU9w68rHT93hY1EhK9WQ/axbwMvAjbkUfBOVjItX6msLGOYnyudWLq+2S1ZNbbuea8SuH2oyzzSFCxy1uOowVih/NC5esniXFj/3NMo+N6G91DQ0NDvpSurjWmnLL8wNxl1Bt9WQ/m8cW96UI2zIzG95yl9SeNehamk3xY7ftTFEFaFEz75yJbRqzWT1tUwh3aaL4WZLVU5zxm271LCl+2etB1reua+ezPWHM1RyxwF2VfAiwZE9u9SwpfvOFuwyDGKMw4u95v1W24Klwca0NzzZ5wAufM5pV8bPriR/1AooKx0kQrzr0PIsvvtM0PSGrp5r4pWlaqnOoC3chvP3hTXz82n6JxImg+xA3FduOxTYehJnKeWb8APAQJxEUZCaq0i1J1T2cRAiiBNt9j59PfMYvKJMgETf2x/ir//sf8jlVIsk72TGpKh1IaVXN74lzqxfX2qVwF3oN55nxU9c5xEXi5+W9imQXB3KyB5QVP0o8lrv89IyfhobGfYV3PrGNX/6ut+LRnf5ZH0pjFGxkii8O2s1t2dbSlNKfJWjxG0SJDndZAN15FL/stlMVP078Zkv0JKy0HVxYbTe+vWPNFu7iUriLgviZpgHTYPN1cVKucwByArwpKaC3hbk5Mcbdl3v8wrhSYbm83uZ2NDncxZkx3IUIF+vxizlhV8XspUPR+AAAIABJREFUq2CZBh7a6nLFbxJRVPxsSzOxW1F13NQLSKqITIROAx0vV8GqitMXRVW4Sxizx8uJH7OzhxFZPSuI3yObCOMUH3mhes5PdY4TKRhL9kPHMhqTXV7iriB+g0lU2tCQ3+PbRxN+P5ZpoOWYnPgdCYqfqsT9k9cP8Gcv7uHZG4cAcmvndgXxS9O01upJM34A8MBqC0d+hCPBQsnfh2NK9ZQrUcRexSBOeI1LYcaPahqyY+i5NgyjmvjpGT8NDY37ArZl4h2Pbp31YcwE8ctCmepp0RyL/rhrAlI/JmGiXJRrNAOpQWszKH47faa0r05R44jMzLvr/Cvf/TZ8/7seaXx718oXjY1n/OJc8ZMXdKQqV9mJacZYDHcBgFvZYpYK3AHJ6inM+A0qFJbL6228cjBGmFlN6XjYf2ezhZumwRIc/QjDIE8QVaUtVuHqZhcv3mMzh/MGR4ifbcoZP4mILGKtmxc9zy4EhJzE4691XIzDuJTEyVU5PuOXWT1Jaao4jq94cB32lDk/VYARXZeisjaSOiWngRM/RbKnHFoEsHPYsQxu9aRwIbqfrism40YlFVAEkZ39jMzR86DPp6E04zfOVHeAFaHLGIf53OqFLNnz5mH+vKYpr3VQhbsM/aiw4dNx8ucuhtnIM36uZfLPAtM0sNIqK8iUDrrMM37zbRdqaGhonBMUiF9Nqucy79CdJkj9mESxVvwWAC3y1mdQ/L7xjRexu9rii6MqLGr1vLrVnen2hY6ypopfKM74Ff/GMQ1EWYCQozjHrm52YRrAY7tsBo5eS674ObawcI241ZM2fvwo4QtaeYF8SejykxU/8XxvOv/WcVm5/CRMCuEuhGmv1+6Kx4nFZE41QXwMVSiNvMhfZKE9Lzousz9SgA3Q/DVuCtowORyHhdeQLH1iqmdUCHdRH0fXs/HMlbXaOT8KUyn2+LH3X7RRDoO48XwfwGb8ALXiN1QQP4CRIHpM+juyZ3Y8q1DJsLPi4cW7o0LNAYG660g9JdK0k1V+yJUOe4LKqrJ6ToIY7ZXippZIqBY5H+TZRoCd52IXccez+AyiqHDKPX7y5vCqwjo8rx37PEFvgWtoaCw1xIWOMtXTzJPrNKaDXi8/TAophxqzYXelhY2ui9c+sNL4b1qOha9+fHvq7doLEr9ZIV5XTcNdmOJXTvUEGMGqm/F7eLuHj/1378XTl9cA5MTvFi+tz1M9ybbm2Wy33jYN+FFu9ZTnJS8LlQ5k+aLn55j1ypkKHdfGnQFbVOYzfvmifNr9bPc9HIxDTASlavZUz/oZvw4nIuw14UToFFOOiagMgygnXCfQ4wcA+9JiXZ41dSwDQSHcpfo43v7wJj55/aCyv47OcZXVs5CEKQWOTMNq24FjGUridzSJCqoyoeVa/Hjo77Z7jHB1hcCZwSTixFJl9SQb5h5X/Nhtqmb89jJS9fBWFzf2J4XZRnoM+swiB8SBULtR5QxogpZjwjSKxzQOpRk/oV6DXoOVll3oHFT1Z660bUWBe8Lvc1mhv9U1NDSWGuKXhWqRRYqfDnZpBm71jGJd57AAep6NP/+Hfxlf9dh0IjcrOgvO+M0KcSe+1WDB45UUv+K151gmooSlIVadYytCQAq3emb2sI6bWz1pkU+PQY9dHe6Sl7hHsRTuMmOqJx0LL2uXFD/PNmFOUc15iMfAn3t+SCTjauJ3HhQ/6jyMF1ro12GtzUiFPOcnP15u9Zxuef2Kq+uIkxSfvHbAfzYKIvzEb3+OBwsBxXO8rSB+Qz/m6ZpNYJoGtnrqLr+BH6GvUvyEWo9bRz5c28RKO7eFi4ofnXdqq2dR8Rv5RaunHHZDBPGpS6sI4oRfDwRxxm9NofhNU17rYBhGSZkrhbu4Nn8OdI2td93yXKB03SkVv7BM9JcNy3vkGhoaGih+Wai+wA3DyIbbl3eH7jRBCswk1Kme5xW0iD6tORNxBqpJF6ZrWwVFRan4ZeEuTVRlmpekGb+Oa/PrmRZmFHDiORb8KKlMUXxgtQ3DYIpfXbhLUzVMJH50nJTq2eQzhxbTt498vhCdVfFrT1X88vRRYDFr3bzIE1AjYebuZKyeNJtGyFW5fMYvitOp4S4A8NTFVQDAp1855D/77U/fxP/67z+P3/vc7Zz4Ce9BR9GbOA5i/vOm2O57XOUWMfArFD/b4qrx9f0xLq62eKAZVRqkacqsnhnxU6V6kuJHr+OgZPUs/g1ZPV9/cYU/tohJkJMqrsqOFFbPOc8HRtCKSaXipljHzVM96fmudVz4Ucx7EkVySqia8aNy+GWFJn4aGhpLjWnhLgBb2C2zJ/804XCrp57xO6+ghXz3lIgfXVemoe48k8FUt1gZfAGQ4sdmrJpsLvBUz4Go+LFjOsgWpwXFL4pzxU9SRV076/LbG5fCXQozfg0XoV3PLsweArni1+Qzh5SXW0e54kdl103hFWb8asJdyOrJycopEr/sfRj60YkRz0qrp1Rub5PVk4e7VJ+D230PO30Pz97IFb+Pv8z+/cU7w9zOLDwXsd+RMAyixuXt/LF7FcRPkeoJsPeZzqHre2NcWs+Te0nxo35LCk5SEb9DbvXMFL/sNptdt2SrBHKC+PpLjCTLxI9ZPdnrQ5siBeK3oAK92nH5MaRpmgUt5a9127V4MilX/DoOkjQPOpqEZavnattRpnouc7ALoImfhobGkkNc6FSlszmWqVM9G4IWv36kLtfWOHvQoqZ9WlbPbGHcdKebZvyqdvJti8Jdqq2eIig9U1XncKCyekYJTx5ULbYvrbVxfX9UUvzsOcNdiEjlil9zRVaM7Z/MOXtHSoVhQEmkcyJydoofzVoO/fjEUj2pe+2gwupJr6ubWT2bWgxfd3EFn76RK36fuLYPAHj+9lBJouUUVYDsh7NdrzsrHq9lEHFUE+5C59CN/TEureXEr+vZGAUxjnz22qx12AzhSDHjl1s9i4pfz7PR9exCWimQl7e/LptnFisdwqzgnq5XyzSw0rKV4S7z1nusCQTNjxKkafHa67o2gjhBFOehT1QZQ6Ew46Ch1TNIln4TWX+ra2hoLDUKVs8qxc/SVs+myOscYm31PKfoLFjnMCvoump6DdGcnSr4AmCLvzBJEcbqOgcVup7NrZBdweqZz/hZ/L9+mGAYRDzwRcbl9bYy3MUwDH7ON5/xs0v/Fmf8pmGz68IwisRvVkWBusdcy1QSc8s04NkmV3dOasauDjzcxY8WKuyuQ9+zYZlGabEeSOTMsUykaa52VW0YEp66uILnbg0wCWNEcYJPZerf83cGtT1+xRm/2RW/nX4Ld4cBn0Wl5xJEiZL4edmMnx/FuHXk4+KarPhFfEOk12LJuEqrpy8rfhSUZKPn2QrFL8RKy8Z610W/ZReSPVVzq2uCQsfuf7GkzNW2wz8H6L5EWy1/P4QAJVKHSQ1WzfittB34UVKofVApg8sGTfw0NDSWGqY5fbFmm6Ymfg1BM1cTbfU8tzhtq6czI/FzbRN+TY+fY5qI4xRx0nyOVK5I4MRvRDN+Jv8vWT1Vi2OAJXu+cjDBJIxhGEWLpz1jGJScHgjkPX5NFoi2ZWKz6+KWQPxmrnPIHqeOyHVciys18szbaYDCeu6NghNT/CjoY39cNeNX7Gukmcdpit9TF1cRJSk+f3OAz98aYBImWGnZRcVP0eNXTPWcT/FLU/DUWCC3Wapm/NoOU/xe2Wcq4SWJ+A39GIMspbLr2iz0RFXnUFL8cjJF9yNibxRgvevyxxStnhNFN+Vap6ik3RsyJZ/6O2fFWsfJg2gEkkrgYTt+zIkuV/zC3AIqB1etUD3IRAyOibTip6GhoXHWcIUdbxXe8tA6nrm8epqHtLSgRZEfJce+I69xPGifutWTiF+z88GzLQRRokw8BKjOIUEUN7N6ArmNsu1YME0W2ORaZqXVs64w+/J6G3GS4treuFDhAOQzrrOkevJjlBS/JkE4AHh64zgrl27NSIhoUV1HVtkiX7J6nqLid2Wjg5Zj4jOvHJ3o46+1ncpUT9HqCeRpldM+557KQkuevXGAj7/MbJ7/4dMXcXcY4NbhJLvv8owfzVTS3NmsGzUU/HNLsHtWpdUC7DyYhDFX3IozfjbGYcyDW3otGx3X4vUEIug2h5OI2SN9NjNnmgZT/BRWT6ppYMQvP15VYJGo0AE5sd3s5t17s2Ct7eBwEiJJ0lzxU1yXoyASrJ6Z4pedG5MKqyeAwpyfShlcNuhvdQ0NjaUHJ34VC4mf+bY34W9+5UOneUhLC1JgokTdsaZx9jhtq+esih8tgmmRKm/IOFbW45ekjRW/ruI5txyR+AlWzyjBwI8rXx/q8nvx7rB0jlvWAlbPjJy2HFYm39QStrPSwu2Bj0kUw7GMmWdrW8Jzrz7O3NZXVbNxkrBMA09eWMGzNw7yMI8T2Fha7ZTnsuQ+STqfhwFzNUz7nLuy3kHPs/HpVw7x8WsH6LdsfM2TOwCAz7x6BKD4Wro265MkkjEJ2dzZLHUOQN6bRzUmQN5b2VelemZWz2tE/AozfuzcoICkvueg5VilAvc0TXEohMccjMMsmMbO7kdt9dzIiNSl9Tau743478YK+/Jq2ynMYd4dBOi61twWypW2gzRlr42K+In1GmNu9SwrfiWrZ4teA7EjMNFWTw0NDY2zBi0gTnMH+35Fwfamid+5xGM7PTxzZQ2vv3g6KjYtapvudNPtjyYhXLs8d1aoc2i4+KeFrljI3nKsCsWPFbhXWT1JCfninWFJcSSrc1NSJM5tiYvNXsturJBu9zzcPpxgHJQj5ZuAHme61bM443faXWSvv8RCUk7y8VUR/KpUT4ApQE3Ip2kaeN0DK3j2xiE+cW0fT19exSPbXQDAX2Q1DzLpbgu9eaSQzbpRQ/UJYrJnrvg5pdu3HZbqeX1vDMMALqy2+O9og4JIZNezCt1+hFEQI05SXNlgmyP74xBDP+ZqZce1FXUOAbdOXlpr43AScdWQEz/J6ikqfneHPjZ786l97P6y/sZxwK2elLDLjlkgfkEM08htnLQpoErrVCl+KmVw2aBXSRoaGksPeSdXY36Ir6FO9TyfWO+6eP8PfCVes9k5lcebZ8YPYDvwnuIcsrMC93CGGT9SHDpOcXaHFvnc7uiYPNylyup5cY0tiA8nUekzw+Ezfs2eK9ltDaNo7Xxws8OVxWnY7ntM8VN0iTUB/U0diem4Nld3zsLqCbBZuSM/whduDU7s8SlYSIRMNLni58eN6kmAPNnzs68e4enLa7iy0YFtGvj8TfZc5GoMUWGlKoFZZ/y2eh4MA7h5KFo92fmuCooRrZ7bPU85d0j31WvZhfoHAimKr9lgmyP7I0am8uAiq6T47YlWz2xT5UZm9xyrZvzaLg7GIe/QuzsI5p7vY/eXl8LT4xU3ZHKrJyl7ZKf2wwRJkmISJqVrb1VRNq/rHDQ0NDTOAaZZPTWaQyt+GjLmtXoeTSJlV5xjGQiy2PWmdmJO/ETFz7Z4Mmeu+JHVs1rx82wLu5maIp/j9oxWT66EZLOHhF/57rfhh7/2iUb3sdP3EMYpXj2czKUmcOI3RfHjM34L9qbNC5qV+2g2J3cixM+x+PMj5Kme0oxfEMFtSPCfuriCcRgjSlI8c3kVjmXiNZudSttqx7V5VcIopECV2d5bxzKx0XELil+d1dNz2Ll/Terwo+MBcvWw56lTPSnI5MFNpmjuDcPCtcRqIXLiF0QJhkGMjS4jSZQken2f2T1VSbWrbQdxknL18s7An3u+D8hrPPZHIVe1VaFL48zq2XYtfi5MorxvVJXqCcjhLvNtzpwn6FWShobG0oNbPbVCtTDEHXA946cBzB7u4opWT8U1aZkmD31oqtLTwlOe8SPwVE/bZIvRKfH5pMbJj09Wz1l6/IBy0E7LsRo/N+rye+neaK6+0SZWT9F66M/ZF7goHt/twzINfOylPQAn83ntWiYneoTKVE8/gjuD4kd4+vIaAODhrR4A9plpSp+VbWF+jlIwZ53xAzI1WAh34XUMFVZPgNVMiPN9QK6A3TqawDTYbVVWT7JoilbPURDzDZeuZ3PCBuTJn6T4XSbil3X5qayeIlEDgLvDAFvHoPjtj0P+movXI80HDzOrZ9u1+DXjh4lwjMXzkcg1kW2AEdnTmq0+KehVkoaGxtLD04rfsUHsVWtqg9K4v0HnQfMZP3a7o0mkvCYd0+DFyY0VPx7uki/oxJ13ntjIZ/ziSqsnwJI9VY8/b4/frB1tIoj4Xbs3Xkjxq5uZ6wrR/Wel+LUcC4/t9HCYLaSn9efNAzdLdRXhRwlModxeDHdpegyP7fThWAa2eh4eyGbnaM5PRaBFUkWv+zz1KzsrLWnGL0/llEFk5uahXyJ+4oxf17NhGAbaWdKnCKpyeA0Rv1GAgZCQ23VZnyZ1C97LiB/N+G31PNimgVcOilbPwoyfYKFMkhT3hotZPYlIHmQkFSj2+LW54hfxonayZU+iWBlAA7D31bNNPuOXpqlO9dTQ0NA4D5hW56DRHKL1TSt+GkBOKBrP+FmC1VOxKLZMg89hNZ/xo3CJYriLfIyezdTEYVBt9QTyxMNSuIs1X7jLrPNbIii9MYgTbkGbBY7FUiSbKn58xu8MPi+fEgKJTuLxmeJbJDN+lBRChuhxx0Hc+Bhc28SXv2Yd73hkk9/PwxnxU73uhXCXOWf8AHZuiKmeg0kEwygSG4J4PchWz1zx89HPros6q+eltRYs08DeKMBICHeh+yEb696Q3X49s3qapoHNnos7WXpoXuCev0Y8jGUU4mAcIk7SxayeRCRHQU78FKFLwyDGKIzRdm3uEJiEiXIOkbCSVUUAQJgFUukZPw0NDY0zhp7xOz6IC2Hb1K+nhjjj17DHL7vdYZbqqbo/mv2ZtcC9ENOuJH4ssCJNMUXxy6yeco+fZTB1qCEhoPTARexfpPix+5vvfqhCogqkQKVpCj9KYBhno+g/JVgmT8TqqVL8wriwAUGfcQO/HO5Th1/4jjfjf/nmp/n/P7zNrJ6qTYJCuAspfnOowjt9D3cGPpJslvXIj9Bz7ZK1FChenyXFLztPRfWOnRMRD1kBwNXYlbaDtbaDvVGY2abzGT8gL5LflxQ/gKl+1M1H17lYjr4mKHR3FyxvB9g1384SfkdBxDs+CXRNjYI4S+U0ueLnh3E+h6i49votm78mdQRxmaC/1TU0NJYeus7h+KAVPw0ZdF01JSV0PQ78SLkoti0jJ34NF975YlW0eubXPakw4uM1sXrKip81RTkrH1dZiZwVPS+vfphnxo/+ru64u56NOEkRxAmCKIFrlWs2TgNE/FRzcccBzzYRxEmBzPhRUjgviOyxcJfmr3fHtQuL/oe3yOqpIn42D3XJA0fmU/yiJOWWysEkUto8geL1eVEmfgLppL9vuxaSFAWiTLbGlZaD1Y6D/VGQlc+riZ9s9QQY8btLip/C6rnKZ/ICThC3FqhzALKKiFE2j+hYhXPbNA0+czkKWUIpV/yipNLqSa8DvSZ0Oz3jp6GhoXHGyOscNFFZFKLKJy+KNb40QQvlpjZEul2aqjdjLNPAJJrN6qlU/NzybJuYIlo3U5UTP0nxM82ZlKiOolh+VhiGgZ0+mxubV/G7tN4pqTwi6H7HQcytj2cBCkk5KZupa5lIUyBKJOLnlIlfGKcLHcdG18Vq21HamduFOof5evwAYHeFnRdk9xQVOxlendVTIJ09weoJ5KockM3lWiY828R6x8WrBxMkgnreywgkdflRQAupeEBR8RuHMWzTKCirnPiNQtzNbreI4kf3uZ/VOagIHPVY0oyfxxW/RElOCf2WzcNdVEE1ywhN/DQ0NJYerhDsoLEYdJ2Dhox5FT9ArYY4pslrGGaucxAWsLR4Exfe4r/rFD9SRFR1DrPM2dFivrvAjB+Q2z3ntZH9yne/FT/03ur6CLHEmilgZ7N47bccXN3snNhnNd2vmOwZSM9X3CB07Pk/4wzDwMPbXWVlSccRC9znJwx5ifsk+6/Pw1FkkH2x79lYaRVv03JMkAgmb6KIyZ6HkxArbRb+st5xcC1L55RnWYnM7g0DdFyrcN5u9V3cHviVYSgth6VqFqyeC8z4AYz4UbiL6rrveIyIT8IEbdeCZRpwLKMQ7jJtxk9bPTU0NDTOCWih6VnL/YF8HiDuzFp6xk8D+fXVuMdPWAgrFT9x4d3Y6lm2VCoVP+HfdeEuLcfCdt9TEL/ZFD9a1HYWSPUEgO3eYsSv49pTw10AZm8MJOvjaeP1l1YXCsOpg4r4+VGstHoCiyuPP/zeJ/B33/NY6ec0U5kkKUZ+hI5rzWVtJSX41pGPURDhE9f28aYH15W3pfdYVvsARlIpEIYrfiriNw7Rz0jjmtAh2OUF7vmsIMCsnqLNEwC2uh6CKMGRH7FidIUCt9Z2sT9iVk/DANY7ajLbFGsdBwcjNuOnItgdh6Xair9v2azwXtU1SGBWT1L8osrbLRNO5srT0NDQOEVwq+cCu7caDKIC42jFTwNz9PgVFL/yIsmZY46UFIGNbr7IJIXDU/T5AfWKHwA8sdvHSrt4G8c0ZiJFpmng4mqrNFM1K0jZOalFJS3cR0GMID47qycA/P2vfy1uHU6m33AO0PPyC8Sv+HxF4jdLuIsK73h0S/lz6pGbRCxJcl6iS0rw7SMff/rCHsI4rXxMuj6rLL8dz8ZQUMTomGSr50o2Aygqi12hxw8AhgGFu4QFmyfAFD8AuHPkY1JRf0AKHZXUN531rQKzegbYDFylpZZSVqnAHcgL7+usnistm3cbjgN2Ti37jJ8mfhoaGksP3uOn6xwWRqHAXc/4aSBfHDfv8atX/OzCwrvZOXZhtYV//f3vwOsv5XUAbTdP8swfWwixmKLC/fNv/TKYUsDJ7mqr1G02DR/4wa9aWMHiit8JWTALVs8wPlPF79Jau3YecRHQ+19Q/EI53CV/z0+KAIuv98iP5u55bDkWVlo2bh5OcDgJ4VgG3nxVrfjRuVO1CdF1LdxGXkwupl0SDie54rcubLLwvkqqRshm/PZGQWEzBsiDWu4MAj5TJ2M1C2MxYCw83wcwdZKsnn1F+E3XszDIFEg6Hlb9EtfO7q20HfhRAj+qv90yQRM/DQ2NpYdnmzNFsGtUQ8/4acjY7nswDEa+mqBIxBTEr6D4Nb9mv+w1xQWvqri8aaonUFzYEn70G16HREiEbIK1zuILV1J2iMweNwpWzzNW/E4S3OoZ52TGj+LCe3ScVs8q5KXhMYbB/IofkJW4H/q4vj/Gl11Zr7yvTo3Vk/2+mMwpnhOEo0nEC+pFJa+qzmFvGPBqFEJO/HyMw7jC6ungpXsjJOliHX6E1baDSZhgbxTgwkr5c6rt2HxekZ53y2HVH3WpnkQijyZR7SzgMkETPw0NjaXH173hgaX33Z8XFFI99YyfBoCHtrr4k3/wnkLfXB3cqYrf8WwucKtngfg1C3epvM8zWtSR1fOkHp9ei1EQ8zqH+xH0vGSrZ+WM32kofkFUmzA7DTt9D5+/dYTn7wzxg+8uzxMStvsefvQbXodveOaB2mOSw11Eq+fhOOTBMGvtnCyTYtl2LBiGQPxGYWk+j4jf3Yz4tRUW8bWOg09cCxFECU96XQSUFPrKwQRvek1ZEe16Fu5lCaIdPhtsZT1+Sfb/5eOk1+JwHGIc3B8zfid65RuG8T7DMD5rGMZzhmH8N4rfe4Zh/Fr2+z82DONq9vNNwzA+ZBjGwDCMnzrJY9TQ0Fh+vPHKGv7uex4/68O4L1BYlGurp0aGpqQPKC6mVTN+1jFVhrT4zr2gMBbqHJZnb3u7x1SKkyJ+oq1Prje4n+Apwl2CKCkktTpzhAvNio6gpg39GJ05NiEIO30PX7g9RJoC73hEPd8HsACX7/xLD/FAmNIxZccw3erJfi8SOrqWTJOFxAyDGFGc4HASlsJdNrouDAO4PQgwCWPlOb3WcbE/DnB74C/c4cfujx1rECWVdQ5HGVml4yHFj+YQVb2WNAN8NIn4LGBnyRW/E7vyDcOwAPw0gK8D8DoA32oYxuukm/0tAHtpmj4K4CcA/M/ZzycA/iGA//qkjk9DQ0NDowxd4K6xKCzT4OeRSlGZJ9xFhZZNM35lq2fLMZfq/H1st4evf8MFvOXqxoncPycifnR/K34V4S5Vit9JEb+2wwjDOFP8FiELOyt5x+Mbr6zNfT+kOnal7kkifkGUYBImueLXERU/u/DvoR/h+v4YaYrSjJ9lGtjouMzqWTXjl1kzjyYRNhWW61khqpPKcBfHLv2+5bBUz6ruPwB83vFwEmKcKYNa8avGWwA8l6bp82maBgB+FcA3Sbf5JgC/mP37/wXwbsMwjDRNh2ma/j4YAdTQ0NDQOCUYhsEXzI62emrMCU9Bygjz1DmoIKbz5Y9btLMtC1qOhZ/5tjfh6lb3RO6fWz3DzOp5n874qRQ/uc7BPuVwF6b4LWb1BIC3PLSx0PHSjF+vVZzxIyWL0itJ8SvO+ImhSTYGfoT/4/eeh2MZePdrd0qPtdXzcOfIV/b4Abk1EwA2j0HxE+9PNQMpHn8x3CWpPEZAtHrmM35nGYx0HDjJT8ZLAF4W/v8agLdW3SZN08gwjAMAmwDunOBxaWhoaGjUwDINxEm6VIqJxvmCa5sYBur0SHFDYSHFrybcZZ75vvsZns0KvMdBnBGh5VYtqqDs8QulOgfh/HNPyM7OiV9IM37zn49ks37HI5sLHRORH97jl10/RGiOJswKuZKRKLJw2qZRUIg7noXP3TzC87eH+Na3vKYU7gKwSoc7A1bnoAx36YjE7zhSPUXip65z4P92csWP0jqrqmrycBc241dlCV0mLDVtNQzjewzD+IhhGB+5ffv2WR+OhoaGxn0BsuLpGT+NeeEpglcIxxXu0lYRv2wBt0zzfacBKvDm4S5LrlpUIU/1lK2e+cLfnGJFPg7QPN1LPFwcAAAXAElEQVQ4iFiq5wKK3xsureLCSgvvferCQsfUlsJdbMuEa5nc6nnIFT+H396zTXQ9u0B2uq6Nz90cwDQN/MC7HlU+1lbPq61zEK2ZW8dA/FanED/RasudApniN6mxehIJZlbPeOk7/ICTJX7XAVwR/v9y9jPlbQzDsAGsArjb9AHSNP25NE2/Ik3Tr9je3l7wcDU0NDQ0gFyF0ameGvOCFtTKVM9CZcj85xjt0qt6/JbN6nka6Hj2/V/nYBUVvyRJEcRJaQOCNh9OLNwlIxpHEzZTuchGxMPbPfzRP3g3HlrQBtyV6hwARoIorZIrfkIP3nrHLSWS0rX119/6YGXFCyN+Da2ex1Dn0HNt0MdKW/Fad6TnDBQVvyqrZ9e1YBoU7pIsfZUDcLLE708BPGYYxkOGYbgA/lMAvyHd5jcAfHv2728G8DtpOmOBjoaGhobGsYL6ELXVU2Ne5DN+5YWS2Le5UKonKX6Oyuq5/Au040bHtbIC9zIRul+Qh7tkgSWZ8ienmBLhO7Fwl4xc3JEqBM4ST11cwRO7/YItsu1Y3Op5OGaK34pAytY6Tsk2vdpx0HYsfN87H6l8rK2eh1EQI0nVYSjHbfU0TYOTSVV1hvj6d7KgFxbukmRWT/X7YxgG+i0Hh+OQpX+eg/dxUZzYllg2s/d3AHwQgAXg59M0fdYwjB8D8JE0TX8DwP8F4JcMw3gOwD0wcggAMAzjBQArAFzDMP4jAO9N0/TTJ3W8GhoaGhoMNlf8NPHTmA91ip9lHo/VU8/4zYZ2ZvX072PFjzYaSPGjdE95A4KUwZN6HTzbhGmwEnPgfJyP737tLt792t3Cz2gzABCtnvmxbvU8TgwJ/+V7HsffeNuDtRUvIplTkSqyZrq2eWzq/FrHxd4orKxz4Mfj5sFTlOpZVymx0rZxmBW4VymDy4QTPRPTNP0AgA9IP/tR4d8TAH+t4m+vnuSxaWhoaGioYesZP40FUZfq6VjHY/XsujZs0+AzSQBTEy3T0FZPBSiGP5Bm3u4nyHUOpPxVWT1PqtbCMAx0XJsTv/Og+KnArJ7qcBcA+G+/4bWI4qIR78pGB1c2yoEuIrYFIqUiS33PhmUa2Oq6xxaWQsetSvUUf0b/9hwLflSf6smO1cHRJMQoiLTip6GhoaFx/4GseHrGT2Ne1M/4HY/Vs+1a+LXvfRse3+0Xfr7Wdo7FPna/oeNa2Bsx6+H9avX0pHAXP+tek89D54QVP/z/7d19jFzVecfx3zOv++JdG7AxYBPjxrx0QxVjO5S0KQICDdBIblJamaoNipCIVJDaqq0KtFXaqmqJqhS1VRIpUWhIRAuIJqpb0SYIqCpV5cWkUF5SUpeAgNDwbhts78vs0z/uvTN3xjOzy3ruvbNnvh/Jmt07d8fHPnNm73Ofc56j6P3ZCvyG83J7otY+1dMsWi+XOOeU6RW9bjqD1q1iplk0NXMQWzkk1jUDv/4Zv86iUAeOzPcN/KbHKzp4ZEGzjcXm37GaDec7EQBQmCTjxxo/rFSrqmeXNX4DmuopSTu3HLvh+V2fuUAbproXnRhl49Wynns3ms4X7AbuHcVdWlM9S13Py2qNnxQFG68figLtbuvOhsFYtdzM9B08uqA19YpKA/jcXz/VuvHSK6ga9A2aZN1g98AvCndqlVLz91oyBfXAke7TQxNTY1W9+OZhuUvj06v/c4XADwDQplXxjsAPK9M345e62M7i5sK2k6eWPmkETdTKejvJ+PXYt2y1S7ZqOHaqZ/uFfR6fcePVsn749hFJ7VUlh8lEraxXD0ZZyYNH55sblh+vdKXObvv4SdLNV/54W5GX47W271TPqA3pIDS5GeDefR1iYnqsqkNHF1QuGVM9AQDhKZeo6onj02+NX1vGL9DM0zCaqFf0bryeK9SMnxS9547J+PWo6pnlVM+JWlnz8fq4Yc34TdQqOjwfZ/yOLLQVdjketUpJa8erfadRXjqzsevxlUqmYfYr7pJuSzrY67vGb6yig0fnVa+UCfwAAOGpsI8fjlP/jN/gpnpi+dKbWIda1VOK/m1J4DfXY6pnM/DLdKpnqqDIkGb8xqplHZmL/o8OHZ1vK+xyvE5aU1ty/dwgXXjWBj33+rua6vJ/nfRFW3XP1M2A8Vrv98H0eFXvzC6osehU9QQAhCe5MCfjh5VqrfHLrrgL3pv0RW+oVT2l9sCv13YOyRTPrIu7JIY34xdt4D7fWNSLbx7WuZvWDuy116+p67nX3s0tS7brjBO164xj1/xKUT9XStaW5Uu/J/oWdxmryF06PBfGdg7h3vIBAKxIkoVhjR9Wqr7sjB+XIXlJZ52Cz/g1q3p2384h6w3cpY5Nw4e4qufh+YbueOgF/fDAUV21c/PAXjvZ0mFYgqXxWrlnxm+pNX7p11jtwh35AIAVqbDGD8eptcavf1VP3mL5SV/0Bh34lbtl/IoL/KplG9r/77FqWe7S57/zfX1k23pdNsB1d+vjip39gqo8TdYqbYFbW8avT0A3Pd4K2ocliD0ew/lOBAAUJsnIkI3BStX6FndJLrptYJs3Y2ndKhqGqF4pN6t5FjrVs1qJH4c3WEiC03fnFvQHH58Z6Hg8Od76YFg2r5+s98749S/uElbGbzhzzwCAwjSLuzDVEyu07eQ12rp+smvxDNaQFmNyhKZ6HrOdQ6+qnjlk/CaHtLCL1Grjr1ywRWefMthtUPZ86HT92PrJofn3//7HZ3TiRGvfwOVW9Wyb6jnEQfxyDUdvAACGRrKdAxUXsVK7t2/S7u2buj7XXENKRjlX6WxFyNs5vKeqnjkUdxmWjFc3O7ecoMtmNuo3Lz1r4K990pq6rviJUwf+uit18dknt32ffk/02mtQUtsWF8MybfV4EPgBANokF+ZkZJCFZO++MhnlXE1Uu09zC029UtI7s9HedL2meuaxgftqyPhtO3lKX/nUrqKbUYhlZ/xSW1wMcxC/XOGOfADAijTX+AWcFUBxWvtEEvjlqW2qZ3n1X8D20lbcZT567MzsJRnPasYbuKcfMVzSGb+lNnBvnhdAX/JbHQDQhgtzZIniQcUYH5GqnvVq+xq/SsmOmb2Qxxq/8XgLh8kh3cph1LVl/PoEdNVyqRkYhrDGL9yRDwBYkeZUPAI/ZIDtQorRvoF7uJd/nds5dK0sm1T1zLK4SxwkTAzxVM9Rlu77pdbuJVs6hLDGL9yRDwBYkdYG7vyKwOC13l8EfnlKbyIecsYvXdxldqGhepeL9Vq5pErJVMrw5kNzjV8A0wNDVCq19ldcKpOXbOkQwlRPbkMAANokd8NJyCALbOdQjJHZwL1S0lyjtcavW8bvkzs2a/MJ45m2I8n0hRAshGqsUlJj0Ze8CTUdr/ObCCDjR+AHAGhTKUV3w9lcG1lIMsms8ctXtVxStWxqLHrQ63dr5XJrO4dG98Dv7FOmBr5vXadWxo9L7WFVr5a16Frydx0ZPwBAsColY/N2ZCbJ9PEey994tay5xmLQN3Wi4i7Rxu1Rxq+Yi/Xx5hq/1R8shGqsWpL70uclWzqEsDaWwA8A0OaTOzZry/rJopuBQFE1tjiT9YoU73EXqlq5pPmGa3HRNbvQKGxa67qJquqVkk6ZHivk78fSlntTYHqsovFqOYgbJgR+AIA2M6dNa+a06aKbgUCZReX12Scyf+O1shYWl5HiWMWSQG+usdizqmcepsaqeuC3L9LGqXohfz+WNlYtLWst+y/s3KytgdwMJfADAAC56ra3GrI3USs3NzUPVb0j8BurFneDYdO6bAvI4PiMVcoqLyOLt+N9J2jH+07IoUXZI/ADAAC5SgqNIF8TtYoOzzaKbkammhm/hUXNLjS0Nl6fBXQ6733rNN8IOwPeicAPAADkqlwylanqmbs19YoOHQ17jV+S8ZtdWNRcgVM9Mfx+7+dmim5C7gj8AABArqplo7hLAa6/eJsOHZ0vuhmZas/4EfgBaQR+AAAgV+USgV8Rdm4JY51SP7VyVKlxbmGx0O0cgGHEbRAAAJCrSqnEPn7IROcav6K2cwCGEaMBAADkak29oskak44weK3tHBpM9QQ68KkLAABy9de/fJ6mx6i2iMFrFneZj9f4FbidAzBsCPwAAECuzto4VXQTEKgk43dkvqHGorPGD0jhNggAAACCUCtHl7bJthVM9QRaGA0AAAAIQhLoJdtWEPgBLYwGAAAABCGZ6nkwzvjVmOoJNBH4AQAAIAjJmj6megLHYjQAAAAgCK2MXzzVk6qeQBOjAQAAAEFIAr93mhk/pnoCCQI/AAAABKFV1ZPiLkAnRgMAAACCUC2bJNb4Ad0wGgAAABAEM1O9UmoGfjUCP6CJ0QAAAIBg1CqlVnEX1vgBTQR+AAAACEY640dVT6CF0QAAAIBg1MolvTPLGj+gE6MBAAAAwUiv62OqJ9BC4AcAAIBgpIM9pnoCLYwGAAAABKM948elLpBgNAAAACAY6cAv2dAdAIEfAAAAApIEe7VKSWZWcGuA4UHgBwAAgGAkGT+meQLtGBEAAAAIRr0Z+FHRE0gj8AMAAEAwyPgB3TEiAAAAEIxm4MdWDkAbRgQAAACCkWT6qOgJtGNEAAAAIBhJwFevssYPSCPwAwAAQDCSgI81fkA7RgQAAACC0cz4EfgBbRgRAAAACEaN7RyArgj8AAAAEAyqegLdMSIAAAAQjOZUT6p6Am0YEQAAAAhGkukj4we0Y0QAAAAgGK3iLqzxA9II/AAAABCMVnEXLnOBNEYEAAAAglEn8AO6YkQAAAAgGK2qnkz1BNII/AAAABCMZG0fGT+gHSMCAAAAwUgyfjUCP6ANIwIAAADBaFX15DIXSGNEAAAAIBitqp6s8QPSCPwAAAAQjK3rJ7V7+2n60NYTi24KMFQqRTcAAAAAGJSxall/uee8opsBDB0yfgAAAAAQOAI/AAAAAAgcgR8AAAAABI7ADwAAAAACl2ngZ2aXm9mzZrbfzG7s8nzdzO6Kn3/YzM5IPXdTfPxZM/tYlu0EAAAAgJBlFviZWVnSFyRdIWlG0tVmNtNx2rWS3nL3bZJulfS5+GdnJO2R9AFJl0v6Yvx6AAAAAID3KMuM3/mS9rv7c+4+J+lOSbs7ztkt6fb463skfdTMLD5+p7vPuvsPJO2PXw8AAAAA8B5lGfhtkvRi6vuX4mNdz3H3BUkHJJ20zJ8FAAAAACzDqi7uYmbXmdk+M9v32muvFd0cAAAAABhKWQZ+L0s6PfX95vhY13PMrCJpraQ3lvmzcvcvu/sud9+1YcOGATYdAAAAAMKRZeD3qKQzzWyrmdUUFWvZ23HOXknXxF9fJekBd/f4+J646udWSWdKeiTDtgIAAABAsCpZvbC7L5jZDZK+Laks6TZ3f9rM/ljSPnffK+mrkr5hZvslvakoOFR83t2SnpG0IOl6d29k1VYAAAAACJlFCbbVb9euXb5v376imwEAAAAAhTCzx9x9V7fnVnVxFwAAAADA0gj8AAAAACBwBH4AAAAAEDgCPwAAAAAIHIEfAAAAAAQumKqeZvaapBeKbkcX6yW9XnQjUBj6f3TR96OLvh9d9P1oo/9H1zD1/RZ339DtiWACv2FlZvt6lVRF+Oj/0UXfjy76fnTR96ON/h9dq6XvmeoJAAAAAIEj8AMAAACAwBH4Ze/LRTcAhaL/Rxd9P7ro+9FF3482+n90rYq+Z40fAAAAAASOjB8AAAAABI7AL0NmdrmZPWtm+83sxqLbg2yZ2fNm9qSZPW5m++JjJ5rZfWb2P/HjCUW3E4NhZreZ2atm9lTqWNf+tshfxZ8F/2VmO4prOY5Xj77/QzN7OR7/j5vZlannbor7/lkz+1gxrcYgmNnpZvagmT1jZk+b2a/Hxxn7gevT94z9wJnZmJk9YmZPxH3/R/HxrWb2cNzHd5lZLT5ej7/fHz9/RpHtTyPwy4iZlSV9QdIVkmYkXW1mM8W2Cjm42N23p0r63ijpfnc/U9L98fcIw9ckXd5xrFd/XyHpzPjPdZK+lFMbkY2v6di+l6Rb4/G/3d3vlaT4c3+PpA/EP/PF+PcDVqcFSb/l7jOSLpB0fdzHjP3w9ep7ibEfullJl7j7ByVtl3S5mV0g6XOK+n6bpLckXRuff62kt+Ljt8bnDQUCv+ycL2m/uz/n7nOS7pS0u+A2IX+7Jd0ef327pJ8vsC0YIHf/N0lvdhzu1d+7JX3dIw9JWmdmp+bTUgxaj77vZbekO9191t1/IGm/ot8PWIXc/RV3/2789SFJ35O0SYz94PXp+14Y+4GIx+878bfV+I9LukTSPfHxznGffB7cI+mjZmY5NbcvAr/sbJL0Yur7l9T/AwKrn0v6jpk9ZmbXxcc2uvsr8df/J2ljMU1DTnr1N58Ho+GGeDrfbalp3fR9oOLpW+dJeliM/ZHS0fcSYz94ZlY2s8clvSrpPkn/K+ltd1+IT0n3b7Pv4+cPSDop3xZ3R+AHDM5H3H2Hoqk915vZheknPSqhSxndEUF/j5wvSXq/omlAr0j6fLHNQZbMbI2kv5f0G+5+MP0cYz9sXfqesT8C3L3h7tslbVaUuT2n4CatCIFfdl6WdHrq+83xMQTK3V+OH1+V9C1FHww/Sqb1xI+vFtdC5KBXf/N5EDh3/1F8YbAo6StqTemi7wNjZlVFF/53uPs348OM/RHQre8Z+6PF3d+W9KCkDyuaul2Jn0r3b7Pv4+fXSnoj56Z2ReCXnUclnRlX/KkpWuC7t+A2ISNmNmlmU8nXkn5W0lOK+vya+LRrJP1DMS1ETnr1915Jn4or/F0g6UBqWhgC0LFu6xOKxr8U9f2euMrbVkVFPh7Ju30YjHidzlclfc/d/yL1FGM/cL36nrEfPjPbYGbr4q/HJV2maI3ng5Kuik/rHPfJ58FVkh7wIdk4vbL0KVgJd18wsxskfVtSWdJt7v50wc1CdjZK+la8drci6W/d/V/M7FFJd5vZtZJekPRLBbYRA2RmfyfpIknrzewlSZ+VdIu69/e9kq5UtLj/sKRP595gDEyPvr/IzLYrmuL3vKTPSJK7P21md0t6RlFVwOvdvVFEuzEQPy3pVyU9Ga/3kaSbxdgfBb36/mrGfvBOlXR7XJW1JOlud/8nM3tG0p1m9ieS/lPRjQHFj98ws/2KCoHtKaLR3diQBKAAAAAAgIww1RMAAAAAAkfgBwAAAACBI/ADAAAAgMAR+AEAAABA4Aj8AAAAACBwBH4AAHQws4aZPW5mT5jZd83sp5Y4f52Z/doyXvdfzWzX4FoKAMDyEPgBAHCsI+6+3d0/KOkmSX+2xPnrJC0Z+AEAUBQCPwAA+puW9JYkmdkaM7s/zgI+aWa743NukfT+OEv45/G5vxuf84SZ3ZJ6vV80s0fM7Ptm9jP5/lMAAKOqUnQDAAAYQuNm9rikMUmnSrokPn5U0ifc/aCZrZf0kJntlXSjpHPdfbskmdkVknZL+kl3P2xmJ6Zeu+Lu55vZlZI+K+nSnP5NAIARRuAHAMCxjqSCuA9L+rqZnSvJJP2pmV0oaVHSJkkbu/z8pZL+xt0PS5K7v5l67pvx42OSzsim+QAAtCPwAwCgD3f/jzi7t0HSlfHjTnefN7PnFWUF34vZ+LEhfg8DAHLCGj8AAPows3MklSW9IWmtpFfjoO9iSVvi0w5Jmkr92H2SPm1mE/FrpKd6AgCQO+40AgBwrGSNnxRN77zG3RtmdoekfzSzJyXtk/TfkuTub5jZv5vZU5L+2d1/x8y2S9pnZnOS7pV0cwH/DgAAJEnm7kW3AQAAAACQIaZ6AgAAAEDgCPwAAAAAIHAEfgAAAAAQOAI/AAAAAAgcgR8AAAAABI7ADwAAAAACR+AHAAAAAIEj8AMAAACAwP0/zD57IJPbb2UAAAAASUVORK5CYII=\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3gAAAHwCAYAAAD0Es3SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXyU9bn///eVWTKZZJKwBAi7qIiCC4qouGFbT8Va3FvbWrXVY221LrWn1fP9tdae7rY9rVvdK2qr9LgVt6pVFFFRAgLKogYEIWxhy77OfH5/ZBKRzQBz3/ckeT0fjzw6yz33XOHRat9c1+fzMeecAAAAAABdX07QBQAAAAAAMoOABwAAAADdBAEPAAAAALoJAh4AAAAAdBMEPAAAAADoJgh4AAAAANBNEPAAAD2CmT1nZhdm+trdrGGima3K9H0BAGgXDroAAAB2xsxqt3oal9QkKZl+/h3n3N86ey/n3CQvrgUAIJsQ8AAAWcs5V9D+2MyWS7rEOffvba8zs7BzrtXP2gAAyEaMaAIAupz2UUcz+7GZrZX0VzPrZWZPm1mlmW1OPx681WdeMbNL0o8vMrOZZvb79LUfmdmkPbx2HzObYWY1ZvZvM7vNzB7q5O9xYPq7tpjZQjObvNV7p5rZovR9K8zsh+nX+6Z/ty1mtsnMXjMz/n0OAJBEwAMAdF0DJPWWNEzSpWr7d9pf08+HSmqQdOsuPn+UpPcl9ZX0O0n3mpntwbV/l/S2pD6Sfibpm50p3swikp6S9IKkfpK+L+lvZnZA+pJ71TaGmpA0RtLL6devlbRKUomk/pL+W5LrzHcCALo/Ah4AoKtKSbrBOdfknGtwzm10zj3mnKt3ztVI+qWkE3fx+RXOubudc0lJUySVqi0wdfpaMxsq6UhJP3XONTvnZkqa1sn6j5ZUIOk36c++LOlpSV9Lv98i6SAzK3TObXbOzd3q9VJJw5xzLc6515xzBDwAgCQCHgCg66p0zjW2PzGzuJndaWYrzKxa0gxJxWYW2snn17Y/cM7Vpx8W7Oa1AyVt2uo1SVrZyfoHSlrpnEtt9doKSYPSj8+WdKqkFWb2qpkdk379Jknlkl4ws2Vmdl0nvw8A0AMQ8AAAXdW2XatrJR0g6SjnXKGkE9Kv72zsMhPWSOptZvGtXhvSyc+uljRkm/VzQyVVSJJzbrZz7nS1jW8+Kekf6ddrnHPXOudGSJos6Qdm9vm9/D0AAN0EAQ8A0F0k1LbubouZ9ZZ0g9df6JxbIalM0s/MLJrusn25kx9/S1K9pB+ZWcTMJqY/+0j6Xt8wsyLnXIukarWNpMrMTjOz/dJrAKvUdmxEasdfAQDoaQh4AIDu4k+S8iRtkDRL0r98+t5vSDpG0kZJv5A0VW3n9e2Sc65ZbYFuktpqvl3SBc65JelLvilpeXrc9LL090jS/pL+LalW0puSbnfOTc/YbwMA6NKMddkAAGSOmU2VtMQ553kHEQCAbdHBAwBgL5jZkWa2r5nlmNkpkk5X25o5AAB8Fw66AAAAurgBkh5X2zl4qyR91zn3TrAlAQB6KkY0AQAAAKCbYEQTAAAAALoJAh4AAAAAdBNdbg1e37593fDhw4MuAwAAAAACMWfOnA3OuZIdved5wDOzkNoOga1wzp22zXu5kh6QdITazg/6qnNu+a7uN3z4cJWVlXlULQAAAABkNzNbsbP3/BjRvErS4p28d7Gkzc65/ST9r6Tf+lAPAAAAAHRLngY8Mxss6UuS7tnJJadLmpJ+/Kikz5uZeVkTAAAAAHRXXnfw/iTpR5JSO3l/kKSVkuSca5VUpbZzhAAAAAAAu8mzgGdmp0la75ybk4F7XWpmZWZWVllZmYHqAAAAAKD78bKDd6ykyWa2XNIjkj5nZg9tc02FpCGSZGZhSUVq22zlU5xzdznnxjnnxpWU7HCzGAAAAADo8TwLeM65651zg51zwyWdJ+ll59z521w2TdKF6cfnpK9xXtUEAAAAAN2Z7+fgmdnPJZU556ZJulfSg2ZWLmmT2oIgAAAAAGAP+BLwnHOvSHol/finW73eKOlcP2oAAAAAgO7Oj3PwAAAAAAA+IOABAAAAQDdBwAMAAACAboKABwAAAADdBAEPAAAAALoJAh4AAAAAdBMEPAAAAADoJgh4AAAAANBNEPAyoLk1pU11zUGXAQAAAKCHI+BlwJ9f+kBH/vLfcs4FXQoAAACAHoyAlwEFuRElU04NLcmgSwEAAADQgxHwMiARC0uSahpbA64EAAAAQE9GwMuATwJeS8CVAAAAAOjJCHgZUBiLSKKDBwAAACBYBLwMKGBEEwAAAEAWIOBlAGvwAAAAAGQDAl4GJDpGNFmDBwAAACA4BLwMKMht6+DVNtHBAwAAABAcAl4GtAe8akY0AQAAAASIgJcBoRxTQW6YEU0AAAAAgSLgZUgiFlYtHTwAAAAAASLgZUhbB4+ABwAAACA4BLwMScTCqmliRBMAAABAcAh4GZKIRRjRBAAAABAoAl6GFMQY0QQAAAAQLAJehhTGwhyTAAAAACBQBLwMScQiqmUNHgAAAIAAEfAypCA3rMaWlFqSqaBLAQAAANBDEfAyJBELSxLr8AAAAAAEhoCXIYlYRJLYSRMAAABAYAh4GdLewatuZB0eAAAAgGAQ8DIkkcuIJgAAAIBgEfAypGNEs4mABwAAACAYBLwM+WSTFUY0AQAAAASDgJchBeyiCQAAACBgBLwMoYMHAAAAIGgEvAzJDYcUDeeohjV4AAAAAAJCwMugwliYEU0AAAAAgSHgZVBBLgEPAAAAQHAIeBmUiEVUyxo8AAAAAAEh4GVQghFNAAAAAAEi4GUQI5oAAAAAgkTAy6BELKJadtEEAAAAEBDPAp6ZxczsbTObb2YLzezGHVxzkZlVmtm89M8lXtXjh0QsrGrW4AEAAAAISNjDezdJ+pxzrtbMIpJmmtlzzrlZ21w31Tl3hYd1+CYRC6u2qVWplFNOjgVdDgAAAIAexrMOnmtTm34aSf84r74vGyRiYTkn1bckgy4FAAAAQA/k6Ro8MwuZ2TxJ6yW96Jx7aweXnW1mC8zsUTMbspP7XGpmZWZWVllZ6WXJeyURi0iSahjTBAAAABAATwOecy7pnDtM0mBJ481szDaXPCVpuHPuEEkvSpqyk/vc5Zwb55wbV1JS4mXJeyURa5t4ZSdNAAAAAEHwZRdN59wWSdMlnbLN6xudc03pp/dIOsKPerxSkEvAAwAAABAcL3fRLDGz4vTjPEknS1qyzTWlWz2dLGmxV/X4gRFNAAAAAEHychfNUklTzCyktiD5D+fc02b2c0llzrlpkq40s8mSWiVtknSRh/V4rpARTQAAAAAB8izgOecWSBq7g9d/utXj6yVd71UNfitIBzwOOwcAAAAQBF/W4PUUjGgCAAAACBIBL4PyoyHlGCOaAAAAAIJBwMsgM1NBbpiABwAAACAQBLwMS8QiqmZEEwAAAEAACHgZlojRwQMAAAAQDAJehhXGImyyAgAAACAQBLwMo4MHAAAAICgEvAxLxMKswQMAAAAQCAJehiViETp4AAAAAAJBwMuwwry2EU3nXNClAAAAAOhhCHgZlohFlEw5NbQkgy4FAAAAQA9DwMuwRCwsSYxpAgAAAPAdAS/DErGIJHFUAgAAAADfEfAyrL2DV00HDwAAAIDPCHgZVsiIJgAAAICAEPAyrH1Es7qBEU0AAAAA/iLgZRibrAAAAAAICgEvwwrZZAUAAABAQAh4GRaPhhTKMTp4AAAAAHxHwMswM1NBbpgOHgAAAADfEfA8kIiF6eABAAAA8B0BzwOJWIRz8AAAAAD4joDngbYOHiOaAAAAAPxFwPNAYSxMBw8AAACA7wh4HiiMRejgAQAAAPAdAc8DbLICAAAAIAgEPA8kYhHVNrXKORd0KQAAAAB6EAKeBxKxsJIpp/rmZNClAAAAAOhBCHgeSMQiksSYJgAAAABfEfA8kIiFJYmNVgAAAAD4ioDngfaAV03AAwAAAOAjAp4H2kc0OQsPAAAAgJ8IeB4oymsf0STgAQAAAPAPAc8Dn2yywogmAAAAAP8Q8DzwySYrdPAAAAAA+IeA54G8SEihHKODBwAAAMBXBDwPmJkSsTAdPAAAAAC+IuB5hIAHAAAAwG8EPI8kciOqbmBEEwAAAIB/CHgeKcyjgwcAAADAXwQ8jyRiEVWzyQoAAAAAHxHwPMIaPAAAAAB+8yzgmVnMzN42s/lmttDMbtzBNblmNtXMys3sLTMb7lU9fiuMRTgmAQAAAICvvOzgNUn6nHPuUEmHSTrFzI7e5pqLJW12zu0n6X8l/dbDenyViIVV29Qq51zQpQAAAADoITwLeK5NbfppJP2zbdo5XdKU9ONHJX3ezMyrmvyUiIWVclJdczLoUgAAAAD0EJ6uwTOzkJnNk7Re0ovOube2uWSQpJWS5JxrlVQlqY+XNfklEYtIEmOaAAAAAHzjacBzziWdc4dJGixpvJmN2ZP7mNmlZlZmZmWVlZWZLdIjiVhYklTdwEYrAAAAAPzhyy6azrktkqZLOmWbtyokDZEkMwtLKpK0cQefv8s5N845N66kpMTrcjOikA4eAAAAAJ95uYtmiZkVpx/nSTpZ0pJtLpsm6cL043Mkvey6ya4k7R08jkoAAAAA4Jewh/culTTFzEJqC5L/cM49bWY/l1TmnJsm6V5JD5pZuaRNks7zsB5fta/B47BzAAAAAH7xLOA55xZIGruD13+61eNGSed6VUOQCungAQAAAPCZL2vweqJPdtEk4AEAAADwBwHPI7FIjsI5xiYrAAAAAHxDwPOImSkRC7MGDwAAAIBvCHgeKsyLMKIJAAAAwDcEPA8lYmECHgAAAADfEPA8lMiNsAYPAAAAgG8IeB6igwcAAADATwQ8DyVirMEDAAAA4B8CnofYRRMAAACAnwh4HiqMhVXb1KpUygVdCgAAAIAegIDnoUQsIuek2mbGNAEAAAB4j4DnocK8sCSxDg8AAACALwh4HkrEIpLEUQkAAAAAfEHA81AiRgcPAAAAgH8IeB6igwcAAADATwQ8D9HBAwAAAOAnAp6H2gNeNQEPAAAAgA8IeB4qTI9oVjcwogkAAADAewQ8D+WGcxQN5TCiCQAAAMAXBDwPmZkSsTCbrAAAAADwBQHPY20Bjw4eAAAAAO8R8DyWiEXo4AEAAADwBQHPY3TwAAAAAPiFgOcxAh4AAAAAvxDwPMaIJgAAAAC/EPA8loiFOegcAAAAgC8IeB4rjEVU29SqZMoFXQoAAACAbo6A57FELCxJqm2iiwcAAADAWwQ8jxXGIpLEOjwAAAAAniPgeay9g8dOmgAAAAC8RsDzWKKjg0fAAwAAAOAtAp7HPungMaIJAAAAwFsEPI8xogkAAADALwQ8jxXmtY1oVtPBAwAAAOAxAp7H6OABAAAA8AsBz2O54ZCi4Rw6eAAAAAA8R8DzQWEsTAcPAAAAgOcIeD4oyA2rloAHAAAAwGMEPB/kRcOqb04GXQYAAACAbo6A54N4NKSGFjp4AAAAALxFwPNBPBqigwcAAADAcwQ8H8SjITUQ8AAAAAB4jIDng3g0rLpmRjQBAAAAeMuzgGdmQ8xsupktMrOFZnbVDq6ZaGZVZjYv/fNTr+oJUh4dPAAAAAA+CHt471ZJ1zrn5ppZQtIcM3vRObdom+tec86d5mEdgYtHWIMHAAAAwHuedfCcc2ucc3PTj2skLZY0yKvvy2bx3LAaWpJKpVzQpQAAAADoxnxZg2dmwyWNlfTWDt4+xszmm9lzZjbaj3r8Fo+G5JzU2EoXDwAAAIB3PA94ZlYg6TFJVzvnqrd5e66kYc65QyXdIunJndzjUjMrM7OyyspKbwv2QDwakiTGNAEAAAB4ytOAZ2YRtYW7vznnHt/2fedctXOuNv34WUkRM+u7g+vucs6Nc86NKykp8bJkT+RF2gIeG60AAAAA8JKXu2iapHslLXbO/XEn1wxIXyczG5+uZ6NXNQUlP7dtLxs6eAAAAAC85OUumsdK+qakd81sXvq1/5Y0VJKcc3dIOkfSd82sVVKDpPOcc91uJ5K89IgmZ+EBAAAA8JJnAc85N1OSfcY1t0q61asaskWcEU0AAAAAPvBlF82eLh5lRBMAAACA9wh4Psjr2EWTEU0AAAAA3iHg+SA/lxFNAAAAAN4j4PkgHmkb0awj4AEAAADwEAHPB+0jmg2MaAIAAADwEAHPB9FwjsI5xiYrAAAAADxFwPNJPBoi4AEAAADwFAHPJ/FomF00AQAAAHiKgOcTOngAAAAAvEbA80leNMQxCQAAAAA8RcDzCR08AAAAAF4j4PmENXgAAAAAvEbA8wkdPAAAAABeI+D5JI+ABwAAAMBjBDyfxKMhNbQQ8AAAAAB4h4Dnk3zW4AEAAADwGAHPJ3nRkBpbUkqmXNClAAAAAOimCHg+iUdDksSYJgAAAADPEPB8khcNSxJjmgAAAAA8Q8DzSTyS7uCxkyYAAAAAjxDwfJKf2xbw6poIeAAAAAC8QcDzSfuIZkMLI5oAAAAAvEHA80n7Jiscdg4AAADAKwQ8n+RFCHgAAAAAvEXA80l+LrtoAgAAAPAWAc8njGgCAAAA8BoBzyd5UY5JAAAAAOAtAp5P4qzBAwAAAOAxAp5PwqEcRUM5BDwAAAAAniHg+SieG2KTFQAAAACeIeD5KB4J0cEDAAAA4BkCno/yoiE2WQEAAADgGQKej+LRMCOaAAAAADxDwPNRPBpSHR08AAAAAB4h4PkozogmAAAAAA8R8HzEiCYAAAAALxHwfMQmKwAAAAC8RMDzEWvwAAAAAHiJgOejeDRMBw8AAACAZwh4PopHQ2pOptSaTAVdCgAAAIBuiIDno3g0JEmqb6GLBwAAACDzCHg+yksHPMY0AQAAAHjBs4BnZkPMbLqZLTKzhWZ21Q6uMTO72czKzWyBmR3uVT3ZID8aliTVNXFUAgAAAIDMC3t471ZJ1zrn5ppZQtIcM3vRObdoq2smSdo//XOUpL+k/7Nbau/g1dPBAwAAAOCBTnXwzCzfzHLSj0ea2WQzi+zqM865Nc65uenHNZIWSxq0zWWnS3rAtZklqdjMSnf7t+gi2tfgNbAGDwAAAIAHOjuiOUNSzMwGSXpB0jcl3d/ZLzGz4ZLGSnprm7cGSVq51fNV2j4EdhtxOngAAAAAPNTZgGfOuXpJZ0m63Tl3rqTRnfqgWYGkxyRd7Zyr3pMizexSMyszs7LKyso9uUVWyIu0TcTWswYPAAAAgAc6HfDM7BhJ35D0TPq1UCc+FFFbuPubc+7xHVxSIWnIVs8Hp1/7FOfcXc65cc65cSUlJZ0sOfvk59LBAwAAAOCdzga8qyVdL+kJ59xCMxshafquPmBmJuleSYudc3/cyWXTJF2Q3k3zaElVzrk1naypy8njHDwAAAAAHurULprOuVclvSpJ6c1WNjjnrvyMjx2rtrV675rZvPRr/y1paPqed0h6VtKpksol1Uv61u7+Al1JPH1MQkMzI5oAAAAAMq9TAc/M/i7pMklJSbMlFZrZn51zN+3sM865mZJsV/d1zjlJl3e+3K4tL9LWwatrooMHAAAAIPM6O6J5UHqDlDMkPSdpH7V157AbQjmmWCSHYxIAAAAAeKKzAS+S3jDlDEnTnHMtkpx3ZXVf8WhY9YxoAgAAAPBAZwPenZKWS8qXNMPMhknaoyMPerq8SIhdNAEAAAB4orObrNws6eatXlphZid5U1L3Fo+G1EDAAwAAAOCBTnXwzKzIzP7Yfti4mf1Bbd087KZ4blh1BDwAAAAAHujsiOZ9kmokfSX9Uy3pr14V1Z3FIyGOSQAAAADgiU6NaEra1zl39lbPb9zqbDvshng0pLXVLUGXAQAAAKAb6mwHr8HMjmt/YmbHSmrwpqTuLY81eAAAAAA80tkO3mWSHjCzovTzzZIu9Kak7i0eDamOEU0AAAAAHujsLprzJR1qZoXp59VmdrWkBV4W1x21nYNHBw8AAABA5nV2RFNSW7BzzrWff/cDD+rp9jgmAQAAAIBXdivgbcMyVkUPEo+G1Jpyam5NBV0KAAAAgG5mbwKey1gVPUhetG0qtp51eAAAAAAybJdr8MysRjsOciYpz5OKurn8aEiSVN+cVHE84GIAAAAAdCu7DHjOuYRfhfQUeVsFPAAAAADIpL0Z0cQeiKdHNNloBQAAAECmEfB8Fk938DgLDwAAAECmEfB81j6iSQcPAAAAQKYR8HyW37GLJgEPAAAAQGYR8HwW79hkhRFNAAAAAJlFwPNZx4hmCx08AAAAAJlFwPNZxyYrTQQ8AAAAAJlFwPNZLBySmdTAiCYAAACADCPg+Swnx5QXCbHJCgAAAICMI+AFIB4NqZ41eAAAAAAyjIAXgLxoSPVNjGgCAAAAyCwCXgDikTAjmgAAAAAyjoAXgHhuiGMSAAAAAGQcAS8A8SibrAAAAADIPAJeAPIiYdWxBg8AAABAhhHwAhCPMqIJAAAAIPMIeAHIz2VEEwAAAEDmEfACkBcJq4GABwAAACDDCHgBaNtkpVXOuaBLAQAAANCNEPACkBcNKeWkptZU0KUAAAAA6EYIeAGIR0OSxDo8AAAAABlFwAtAfjQsSapv5qgEAAAAAJlDwAtAXrqDx0YrAAAAADKJgBeA9hHNOgIeAAAAgAwi4AUgr2MNHiOaAAAAADKHgBeA9jV4jGgCAAAAyCQCXgDYRRMAAACAFwh4AWBEEwAAAIAXPAt4Znafma03s/d28v5EM6sys3npn596VUu2iXcck0AHDwAAAEDmhD289/2SbpX0wC6uec05d5qHNWQlRjQBAAAAeMGzDp5zboakTV7dvyvLDecox9hkBQAAAEBmBb0G7xgzm29mz5nZ6IBr8Y2ZKR4N08EDAAAAkFFejmh+lrmShjnnas3sVElPStp/Rxea2aWSLpWkoUOH+lehh/KiITZZAQAAAJBRgXXwnHPVzrna9ONnJUXMrO9Orr3LOTfOOTeupKTE1zq9Eo+G6OABAAAAyKjAAp6ZDTAzSz8en65lY1D1+I0RTQAAAACZ5tmIppk9LGmipL5mtkrSDZIikuScu0PSOZK+a2atkhokneecc17Vk23yoyHVNTGiCQAAACBzPAt4zrmvfcb7t6rtGIUeKRELq7K2KegyAAAAAHQjQe+i2WMlYhHVNNLBAwAAAJA5BLyAFMTCqiXgAQAAAMggAl5AErEwHTwAAAAAGUXAC0hhLKLmZEqNLeykCQAAACAzCHgBScTa9rehiwcAAAAgUwh4Afkk4LUEXAkAAACA7oKAF5BEbkSSVMtZeAAAAAAyhIAXkAJGNAEAAABkGAEvIIxoAgAAAMg0Al5ACmNtI5rVdPAAAAAAZAgBLyDsogkAAAAg0wh4ASnIZUQTAAAAQGYR8AISDuUoHg2plg4eAAAAgAwh4AWoIDfMiCYAAACAjCHgBSgRC6umiRFNAAAAAJlBwAtQIhahgwcAAAAgYwh4AUrEwhyTAAAAACBjCHgBKoxFVMsumgAAAAAyhIAXoESMTVYAAAAAZA4BL0DsogkAAAAgkwh4AUrEImpoSaolmQq6FAAAAADdAAEvQIlYWJI47BwAAABARhDwAtQe8BjTBAAAAJAJBLwAJWIRSeKwcwAAAAAZQcALUCEdPAAAAAAZRMALUEcHj4AHAAAAIAMIeAEq6OjgMaIJAAAAYO8R8ALEJisAAAAAMomAF6COYxKaCHgAAAAA9h4BL0C54ZCi4RxVM6IJAAAAIAMIeAErjIUZ0QQAAACQEQS8gCViEQIeAAAAgIwg4AWsIDfMLpoAAAAAMoKAF7AEI5oAAAAAMoSAF7BELKxaAh4AAACADCDgBaxtDR4jmgAAAAD2HgEvYIxoAgAAAMgUAl7AErGIaptblUq5oEsBAAAA0MUR8AKWyA3LOam2mS4eAAAAgL1DwAtYIhaWJDZaAQAAALDXCHgBS8QiksQ6PAAAAAB7jYAXsPYOHjtpAgAAANhbBLyAFea1dfCqGgh4AAAAAPaOZwHPzO4zs/Vm9t5O3jczu9nMys1sgZkd7lUt2ayYgAcAAAAgQ7zs4N0v6ZRdvD9J0v7pn0sl/cXDWrJWEQEPAAAAQIZ4FvCcczMkbdrFJadLesC1mSWp2MxKvaonW7WPaG6pJ+ABAAAA2DtBrsEbJGnlVs9XpV/bjpldamZlZlZWWVnpS3F+CeWYCmNhOngAAAAA9lqX2GTFOXeXc26cc25cSUlJ0OVkXFE8QsADAAAAsNeCDHgVkoZs9Xxw+rUepzgvqi31zUGXAQAAAKCLCzLgTZN0QXo3zaMlVTnn1gRYT2CK4xFtoYMHAAAAYC+FvbqxmT0saaKkvma2StINkiKS5Jy7Q9Kzkk6VVC6pXtK3vKol2xXmRVSxpSHoMgAAAAB0cZ4FPOfc1z7jfSfpcq++vyspzouoil00AQAAAOylLrHJSnfXPqLZlnkBAAAAYM8Q8LJAcV5UyZRTXXMy6FIAAAAAdGEEvCxQ1HHYOTtpAgAAANhzBLwsUBRvD3iswwMAAACw5wh4WaA43cGr5qgEAAAAAHuBgJcFOjp4BDwAAAAAe4GAlwWK86KSGNEEAAAAsHcIeFmguKODxyYrAAAAAPYcAS8LxCIhRcM5qmJEEwAAAMBeIOBlieK8iKoY0QQAAACwFwh4WaI4HmENHgAAAIC9QsDLEkV5EUY0AQAAAOwVAl6WKMqLckwCAAAAgL1CwMsSxfGIqurZRRMAAADAniPgZQlGNAEAAADsLQJelijOi6iuOanm1lTQpQAAAADoogh4WaL9sHO6eAAAAAD2FAEvSxTmEfAAAAAA7B0CXpYojkclSVUNbLQCAAAAYM8Q8LJEcbqDx2HnAAAAAPYUAS9LFDGiCQAAAGAvEfCyRPsmK3TwAAAAAOwpAl6WSMQiMpO20MEDAAAAsIcIeFkilGNK5IZVTcADAAAAsIcIeFmkOB7Vlnp20QQAAACwZwh4WaQ4HmFEEwAAAMAeI+BlkaK8CJusAAAAANhjBLwsMqg4Tys21sk5F3QpAAAAALogAl4WGTUgoc31LVpf0xR0KQAAAAC6IAJeFhlVWihJWrymOuwxFcoAACAASURBVOBKAAAAAHRFBLwscuCA9oBXE3AlAAAAALoiAl4WKYpHNLAopiVr6eABAAAA2H0EvCwzqrRQS+jgAQAAANgDBLwsc2BpQksra9XUmgy6FAAAAABdDAEvy4waUKjWlFP5+tqgSwEAAADQxRDwssyBpQlJYkwTAAAAwG4j4GWZ4X3ylRvO4agEAAAAALuNgJdlwqEcjeyf0JK1dPAAAAAA7B4CXhYaNSDBUQkAAAAAdhsBLwsdWFqoDbXNWl/TGHQpAAAAALoQAl4WGsVGKwAAAAD2AAEvCx04oFCSGNMEAAAAsFs8DXhmdoqZvW9m5WZ23Q7ev8jMKs1sXvrnEi/r6Sp65Uc1oDBGBw8AAADAbgl7dWMzC0m6TdLJklZJmm1m05xzi7a5dKpz7gqv6uiqRpUmtIijEgAAAADsBi87eOMllTvnljnnmiU9Iul0D7+vWzmwtFBLK2vV3JoKuhQAAAAAXYSXAW+QpJVbPV+Vfm1bZ5vZAjN71MyGeFhPlzJqQEItSadlG2qDLgUAAABAFxH0JitPSRrunDtE0ouSpuzoIjO71MzKzKyssrLS1wKDcmBp20YrixnTBAAAANBJXga8Cklbd+QGp1/r4Jzb6JxrSj+9R9IRO7qRc+4u59w459y4kpIST4rNNiP65isaymGjFQAAAACd5mXAmy1pfzPbx8yiks6TNG3rC8ysdKunkyUt9rCeLiUcytH+/Qu0eC0BDwAAAEDneLaLpnOu1cyukPS8pJCk+5xzC83s55LKnHPTJF1pZpMltUraJOkir+rpikYNKNSMD3vGSCoAAACAvedZwJMk59yzkp7d5rWfbvX4eknXe1lDV3ZgaUKPzV2lDbVN6luQG3Q5AAAAALJc0JusYBfaN1p5nzFNAAAAAJ1AwMtiowYkJLGTJgAAAIDOIeBlsT4FueqXyNVidtIEAAAA0AkEvCw3qrSQDh4AAACATiHgZblxw3pp8dpqrd7SEHQpAAAAALIcAS/LnXHYIDknPTmv4rMvBgAAANCjEfCy3NA+cR05vJcen1sh51zQ5QAAAADIYgS8LuDMsYNVvr5W71WwFg8AAADAzhHwuoAvHVyqaDhHj81dFXQpAAAAALIYAa8LKIpH9IUD++mp+avVkkwFXQ4AAACALEXA6yLOHDtYG+ua9b8vfqDNdc1BlwMAAAAgCxHwuoiJB5ToxJEluv2VpTrq1y/px48u0PrqxqDLAgAAAJBFCHhdRCSUoynfHq/nrjpe5x4xWE+8U6HP/eFV3T1j2U7HNjfXNevOV5cy1gkAAAD0EAS8LubA0kL98syD9cI1J2j8Pr31y2cX68zbX1f5+trtrv3zSx/q188t0fQl6wOoFAAAAIDfCHhd1PC++brvoiN1x/mHq2Jzg0675TX97a0VHe9vqG3SI7M/liT9a+HaoMoEAAAA4CMCXhd3yphSPX/1CRq/Tx/9vyfe0yNvt4W6e2d+pKbWlI4c3kv/XrROza2MaQIAAADdHQGvG+hXGNN9F47TCSNL9JN/vqeXFq/Tg2+u0KkHl+rSE/ZVdWOrZi3bGHSZAAAAADxGwOsmwqEc3XLeWA3uFdfFU8pU29Sqyyfup+P376t4NMSYJgAAANADEPC6kaJ4RHdfME6JWFgnH9RfBw0sVCwS0kmj+umFhWuVTDmt2Fin/3l6EUcsAAAAAN0QAa+b2a9fgV754UTd8rWxHa9NGjNAG2rbDkk/7ZaZunfmR7p4Spnqm1slSeXra3TZg3M0f+WWjNdTWdOk3z//vjbWNmX83gAAAAA+jYDXDfUpyFUsEup4PvGAfoqGc3Tr9HIN6RXXL88co4Wrq3TVI/P00uJ1OvO2N/SvhWt18ZQyrdpcv939Zi/fpEfe/ljOud2qo6qhRRfc97ZunV6u7/5tLufxAQAAAB4LB10AvFeQG9blE/fT5vpmXTdplGKRkFqTTjdMW6gXF63TQaWFum7SKF3+97m6ZEqZ/u+yY5SIRbS+plG/eXaJHn+nouNe540f2qnvbGhO6pIps1W+vkbfOna4/vr6ct341EL94oyDvfo1AQAAgB6PgNdDXPWF/T/1/MIJw1XV0KI1VQ36yWkHKR4N6y/fOEIX/vVtfeGPryqZctpY16xITo6uOGk/zVu5RTdMW6jDhhZr1IDCXX5XSzKly/8+V2UrNuvm88bqy4cOVDScoztfXaYDSwv1jaOGdVz7r/fWqnx9ja743Cf1NbYktbaqUcP75mf2DwEAAADo5mx3x+6CNm7cOFdWVhZ0Gd3WMwvW6Ml5FepbEFW/REynHzZQI0oKVFnTpFNvfk2JWFjTrjhOBbk7/ruBVMrp2v+bryfeqdAvzhij849uC3PJlNO375+t18s36O//ebTG79Nb766q0tl/eUPNyZQeufRoHT2ijyTpOw+W6aXF6/X49ybokMHFnvyeNY0tuuGfC/W1o4bqyOG9PfkOAAAAwAtmNsc5N26H7xHw0FlvLN2g8+95S8XxqE47pFTnHDH4UwHMOacbn1qk+99Yrh/+x8hPdeWktjV5Z9z2uqobWvTQJUfp0gfL1Jp0ck7qXxTTk9+boFc/qNRFf52tcI5paO+4nr7yOMWjbWEymXIK5din7vleRZU21jWroTmpQwYXaWBx3mf+Hs45XTN1np6ct1p9C6J65srj1b8wloE/IQAAAMB7uwp4bLKCTpuwb189dMlRmrBvH02dvVKTb31dNz2/RMmUU21Tq679x3zd/8ZyXXzcPrr8pP22+3xRXtsxDs2tKX35lplas6VRt379cF37HyM1f+UWPTmvQj+btlAj+ubr7gvHadmGOv3q2cV6d1WVvn73LB3+Py9q3VbHOzw0a4VOu2WmLrzvbV320Bxd9cg7nfo9HptboSfnrdZXxw1RXVNS33/4HbX6sAHMrGUb1dCc9Px7AAAA0HPRwcMeqWls0a+eXayH316pY0b00ZqqBn28qV5Xfn5/Xfm5/ZWzTadtay8vWafLHpyr6yaN0reP20fJlNOXbn5N5etr1ZpyeuDb43XCyBL98plFuvu1jyRJvfOjqm5o0flHD9PPJo9WU2tSJ/7uFZUWx/T/fekgPb1gte5/Y7lmXf/5XXbjllbW6su3zNQhg4v0t0uO1rT5Fbpm6nx946ih+u7EfTW4V7xTv/97FVWKR0MaUVLQqetfWrxOF08p09fGD9GvzzqkU58BAAAAdoQOHjIuEYvo12cdot+dfYjmfLxZTa0pPfyfR+vqL4zcZbiTpM+N6q/5N/yHvn3cPpKkUI7px5NGqTXldMroATphZIkk6YdfPEBfPnSgrjhpP736XxN1zhGD9fe3PtaaqgY9OmeV1lY36gcnj9QRw3rp6+OHyjnp+YVrO75n6uyP9YunF2lTXbMkac6KzTrvrlnKDefoT18dq1CO6cyxg3XBMcP0t7c+1nG/na4Tfjddd81YqubWnXf0KmuadN5ds/SNe97qOEtwVxpbkrrxqUUyk6bOXqny9bWf+RkAAABgT9DBw15bvaVBiVhYiVhkj+/hnNMr71fq8GG9VJS34/us3FSvk37/is4dN1ivfbhBfQty9cT3JsisLVB+/g+vqF8ipocvPVoba5t03G+nq6ElqaK8iCYfOlCPzP5YpUV5uvuCcTpgQOJT3/3+uhq9uXSjXly0Tm8s3agRJfm65LgRWrW5Xu+vrdEpYwbo3HFDJEnXPbZAj85ZpdaU0/cm7qsfnTJql7/brS9/qN+/8IH+fN5h+n9PvKfj9uurO755hBpbkrrntWWaeEA/jRlUtMd/dl549YNK5UVCGr8PG9AAAABkm1118DgmAXutMxubfBYz00mj+u3ymiG94zp33BA9/PbHkqSfnz66I9xJ0qkHl+q26eXaWNuke2d+pMbWpO44/3A98OYKPThrhY7fv69u+dpYFcej2333qAGFGjWgUN86dh9NX7JeNz61UP/9xLsK55j6FET10pL1ioRytH//Ak0tW6mLj91Hm+tbdPdry3TOEYM1oqRAyZSTSZ/qYFZsadCt08s1acwAnX7YIK3YWK8/vviBnnhnle6a8ZEWr6nWvTM/0mPfndDpcU+vbaxt0nfSG+D88auHafKhA9XcmtJDs1aoT0FUpx82aLvPNLUm9ad/f6gzDhv0qfAMAAAAf9HBQ5dSsaVBE2+argMGJPTUFcd9KuAtXF2lL908U9dPGqVbXi7XiQeU6LavH97Rodu/X2K7XTh3pqk1qeUb6jWsT9uavG/9dbZmL9+koX3i2lzXrFf+6yQ1tSb1+d+/qoMGFmrUgISeWrBGOSZdeMxwnTNusJ5/b63umrFMm+qb9e8fnKjBveKqa2rViTe9og21TeoVj+jHp4zSTc+/r/zcsB7/3gT1LcjdZV2plFNDS1L5OzmmorMenLVC0+ZV6O4Lxm0XeH///Pu67ZVyHTKoSAsqqvSdE/bVCwvXatmGOuWGczTjRydtt87x/tc/0s+eWqTBvfL09PeP2+6eAAAAyBzW4KHbGFScp/suOlJ/Pm/sp8KdJB1UWqihveP6/Qvvq7apVVekd/Js79B1NtxJUm44pAMGJBSLhBSLhHTXBUfogAEJLaus0w/+4wAV5UXULxHT1SeP1FsfbdLDs1fqmH37aMygIv3hxQ90zK9f1s+eWqSBxXm6/1vjOzZvyc8N6xdnjNYXR/fXM1cer/PGD9U9F47T+ppGXTylbKdr+qbO/lhfueNNHXLjCzr4Z8/r3pkfdbzX2JLUv95bq/U1jTv87Lbuf/0j/eTJ9zR7+Wbd+NSiT71X3diiKW8u16QxAzT1O8do4sgS3fHqUqWc02/PPljJlNNt08s/9Zm6plbdOr1cI/sXaF11o66ZOk+plNP8lVt0+d/n6uUl6zr9557NttQ379b1zjnVNX32Gs3P0tyaUmVN017fBwAA9Ax08NCt/PrZxbpzxjKdfFB/3X3BDv9SY49tqmvWy0vW64zDBiocavu7kVTK6Y2lG3XIkCIVptcgLllbrWffXavj9+/b6UPUX1i4Vpc9NEefG9Vfd37ziE+F0TkrNumcO97UyH4JHTWit1ZtbtDLS9br2pNH6sQDSvSDf8xX+fpaxSI5unDCcJ1/1DD1yo8qLxJSZU2TVmys08a6ZrWmnD5YW6Nbp5fri6P7a79+Bbpt+lLdfcE4nXxQf0nS7a+U63f/el9Pf/84jRlUpObWlN5YukHH7NtHueGQrn/8XT06Z6Wm/3BiR2htX2P4+PcmaGFFlX7yz4U6qLRQi9ZUS5JikRw9etmE7dYZNrUm9eG62h2uP9xc16wfP7ZAfQqie7XraDLlVNXQouK8yGdu/rOre9z80oe6+eUP9ZMvHdSxOVAq5XTnjGU6YWRfjR74ye+wrLJWj8+t0NMLVmv5xnodMrhIJx/YX2cfMXi3x5lTKadLHijTy0vWa/w+vXXm2EE6c+wgxSKhHV7vnNvuLz4AAMCemTr7Y81fVaUbJ49WJJRdfTEOOkeP8cG6Gl1w79u696Jxn/o/3V3BA28u10//uVDfPHpYx/rCxpakvnTza2psSen5a05QQW5YrcmUfvTYAj0+t0JmUv9ETD+edIBmfLBBT86r0Gf9T/qU0QN0y9fHyjnp9Nte14baJj108VFycvrG3W9pzKAiTfn2+B1+dvWWBk286RWddfgg/ebsQ7S5rlkn/G66jhrRR/dcOE7OOf3Xowv07Ltr9J/Hj9CZYwfp63fPUspJ/7zi2I7RzvcqqvSDf8zTB+tqdd2kUbrsxH07vmPR6mpd+mCZVm1ukCT94zvH7NZmL7VNrbrl5Q/1xNwKbahtUspJB/RP6IbJB2nCvn21pqpB0+at1vKNdappbFWOmW6cPFq98tvGSptak7p35keKhUMqLYrpobdW6PXyjeqdH1VrMqVX/+sk9cqP6uG3P9b1j7+rPvlRPXn5sRrSO67Xyzfo2/fPVksypQn79tWhQ4r0evlGzVu5RYncsH511sH68qEDO/273PHqUv3muSWafOhAvbe6Sssq6zR6YKHuuXCcSos+CYuNLUn94plFembBGt11wbhO/8VCZ7QkU3r23TVaur5WZx8xWMP65Gfs3gAAZLMzb39dza0pPXPl8UGXsh0CHtBF/OrZxbprxjKdNXaQLjl+hJ5esFq3v7K042zAdqmU000vvK8t9S267pRRKoq3dQ/L19fo7Y82q7apRXVNSfVN5GpY77j6FeYqnJOjaChHQ3rndXR5Fq6u0um3vq7W1Cf/HPisQPWzaQv14KwVOmJYL1VsbtDqqgb966oTOjZXcc6pOZlSbrity7R4TbXO/ssbKknk6rAhxQrlmKbNW63e+VGNKi3UjA8qdePk0Zp86EDdO/Mj3TNzmYrzovrfrx6mq6e+o0HFeXrsuxMkSXfOWKY3lm7U0SN664T9S7br/j377hrd+NRCratu0qQxA7RfvwLl54b10KwVWrW5QaMGJPT+uho5J/UtyFVhLKzlG+t04YThuuHLoyVJt00v103Pv99xz9xwjv7n9DE6dEixJv15hi6asI8umzhCX/jDqxreN1/LN9RpQFFMP/riKF3x8FwN652vKd8erwFFn6xTXLGxTtdMnae5H2/RWWMH6YtjBmjfkgIN7xPv6AZva86KTfrKnbP0xdH9ddvXD5ckvbhonX7wj/nKi4b0h3MP1cDimDbXt+gnT76nJWtr1LcgqsaWlB665CgdNqR4V/9V+0zOOd3/xnLd+eoyra1uG//NMWnSmFL9+JRRGtqnc2dGBqViS4PuenWpciMhXT9pFJ1NAMBuWb2lQRN+87L+64sH6PL0sp9sQsADuohUyunXzy3Wg7NWqLGl7Sy+c48YrJvOPdSz71ywaos+WNc24tkvEfvMblllTZOu+PtcSVKveFQnjSrRV48cusvPvPpBpf744gfaXNesmsYWnTiyRD+bPFr5uWFd/re5emHROsWjITW0JHXqmFLdMPkg9UvE9MjbH+u6x9/VHecfoYWrq3TLy+UaWBTT6qq2wHHV5/fXNSePlCS9vGSdvn1/mcYMKtTPTx+jw4f26vj+xpak7pqxTC8tXqcTR5borMMHa3jftk5U+9jpv39wovIiIZ30+1c0Yb+++s1ZB2tNVaP6FEQ7umXXPbZAj81dpfH79NbsjzbruauP19qqRl1439tqTTntW5Kvqd85Zoeb5bQkU/rzvz/U7a+Uqz1PF+SGdeTwXho3vLeSKact9S3aUt+szfXNbV2/WERPX3lcx/iv1NalvmRKmT7eVN/xWp/8qP7wlUM1akChvnLnm9pS36zrJh2YPr4krIMGFqpfIrZtSbvUPnp71D69dekJI3TQwEJNeWOFHpq1QiWJXE274tjdPhpl+YY6LVlbo/36FWho77gqtjRo0epqba5v1oDCmAYW5+nA0sQuw1jFlgZd88g89c6P6qdfPmi7sdfqxhb9+tklenTOSiVTTiknXXvySH3/8/vvVq07smJjne6d+ZGu+cLIjo6vl2qbWjV/5RZN2LcPARUAfHbPa8v0i2cW65UfTuz4/wzZhIAHdDFb6pv1f2WrNH/VFv3yjIM7OnTdUVNrUtc99q5Szunyk/bTyP6fHLPQmkzplD+/prVVjaptatV5Rw7Rr848WBvqmvTrZ5foiXcqdMf5R+jQIUU69c+vaUBRnp743oSdrlHbkfXVjTrxplf0uQP7KT8a0hPvVOjFa07c4T/M269taEnq6i/sr6u/0BYun3hnlR6bU6E/fOXQ7XYY3VZNY4uWVdapfH2t3lm5WW8s3ahllXWS2gJfUV5EvfIjKinI1Y9OGaUDSwu3u0dVfYteK6+Uc1IoxzR+n94doXLV5np99c5ZqtjS8KnPDCrO06gBCQ3pHVdpUUy1Ta3aUNusjbVN2pgO3qeMHqCLjx+hl5es0zVT5+ussYP0h68c+qlw8dayjfr6PW/plNEDdOvXt9/saEecc5o6e6VumLZQTa2pXV572iGl+tNXD9thZ/PNpRt1+d/nqqU1pZZUSiEzXXPySJ1/9DDFIiGtqWrQt/46W+Xra/W18UN12cR99fvn39cT71ToL984XJMOLv3MWqsbW5QXCW231qI1mdLZf3lD81dV6eSD+uuubx6xy9/dOafN9S1aW9Wowrxwx5rVznDO6ekFa/TLZxZrbXWjfnv2wR1/idKaTKW72H0UDWd2Pciyylo9995aXTRh+A536q1qaFFrMqU+n7Hbb7aZ+/Fm9S+MaVAGjvQB0HOcefvrampJ6dmrsm88UyLgAejCXly0Tv/5QJnOPnywbjrnkI7NUv7/9u48PKrqfOD492Qmk31PCEkISYCw70vYBNxRqCKtVrBWEKu1Wmtbu1p/rXbRtrZatS7ViuCutS6gKEUEQbYkyA4hJCEL2ZPJOslktvP7Y4aYkAUQSGDyfp4nz8zcuZmczJt7M+897znHandy4/PbySlvICUmiKOVFlbfc9HXWk/wsXXZPLn+CErB7bMGcf+8EV3u++r2Av53sJwXbpnUWoZ6phqsdvyMhrP2gd1qd1JR34LV4aTGYmNfcR27i2rJqWikyNyExeZEKYgMNBEd7EdUsLs3amtuNSH+Rqx2J5OTIlm5LK3TNj27MZe/fJLFg9eMZOnMlNbt2eUNvLq9gLEDwrl8RD/8fQ3sKqzl9fRCVu8p4aIh0dx7eSqF1U0UmJtICPdnVHwY0cF+lNdb+fRQOU99lsM3JyTwtxvGoYE9x2rZnldN+lEzm49UkRwVyPO3TMZk8OGB9/fzeXYlYQG+LJyQwCf7y2hscfDczZO4KDW69b1Y/MJ2DpXWk5YSBUByVCA/uiy1XU/rwZJ6nt+Uy+q9pUQHm7h1ZgqL0wYSFuC+uPLk+iM8ti6by0fE8umhcn6/YBS3TE/u9P0vqLZww3PbqPDMfuqj4LvTkvjpFcMoNDfx3KZcdubX0D/MnXQ0tDg4Zm6iqrEFf18DBh9FaZ2V0QmhGJQir8rC+p/OISbEjwfe389rOwpJS4nkuZsnEdlFT+LBknr+8OFBrhkXz01Tu+9hd7o0/96cx2PrsmlxuJicFMHyW6e06znekVfN3a/vosXh5KnFE7h4WPfrlp5tLpf+WhMl7Sqs4YbnthEZZOI/d07vcgyp1pq3M4vQGhaldf9+9TSbw0Wz3dn6t/h15VdZcLg0Q/qdH2uuCnE+O9/LM0ESPCHEBS6vspHkqKAOH/DK6qx846kvqGps4YlF4ztdhP1UNLY4uPjRDWgNG35+cbsPtt5Ga01Di4Mgk7HD0iEHS+r5x6fZmC02Xlw6pcsPlC6X5vaXM9mYXcmdcwZxz6WpZObX8INXd2KxOXB5ehYNSmFzujD4KH58WSp3XTLkpMuVPLX+CH9fl824xHAKqy3UNNkBGNIvmFmp0fz0iqGtpaFaa7bnmXl1RwFr95cRHezHS7dO6dDrWdFg5f5391HZaAOtOVhaT6DJyM+uHEqLw8VH+0rZVVhLkMnADZMTOVLRwJacagJNBq4ZG09aSiS//O9e5nt6F29bmckXOVW8f9dMRsZ37GG9bUUG2/Oque/KYfQP82dbbjWv7SjAz+guQw7xM3LJ8H5UW1ooqbUS7GckMTKAmGA/Wjwf5tNSIlk0ZSAF1RauemIzlw7rx6SkCP605hCXj4hl05FKYkP9eGLRBCYkhrf2JmqteSO9iAdXHwANNqeLm6cN5HfXdD4D3M4CMw+tPsjeY3VcPiKWS4f347cf7GdkfCh/vX4sDqfmi5wqHl17mKTIQPx8DWSV1fOLucO5c86gLnsxS2qb+U/mMVbvLeEbY+O497LUdvtqrcmpaKTQ3MS0QVGtPYZ5lY2s2VeKzdPTW1JnZU9RLbmVjUxOimThxATmjY47paqGequdeU9sbl0/NNjfyH++P6Pd+Fhw90z+8p29fHKgDKDb5B1ga24Vn2dX8vMrh3XoaS6vt3L7y5kMig7iTwvHtP5eVrsTH6VO6yJOkbmJ19MLeTujiNpmO9eMjePOiwczvH/Hv7mT2VlQw5Ll6ThdmuVLpzB9cFSX+9qdLuqa7Sddl7Wn1VhsvJVZRH6VhUJzEyH+RiYlRTBjcHSnszGfLXani9+vPsiQfsEsmZF8zn6OOL+c7+WZIAmeEMKLZZXVc7Cknm9OHHBGr5NT4Z58JbVNiajoWoPVzoOrDvLfL4+RGBlAaa2VwTHBvLh0MmaLjbUHynA4NWkpkUxOijytMuN/fJrNG+mFzBwczZxhMcwcEn3SD5vVjS0YDT6n1MuRU9HAb97bz46jZgBGxYdyzbh4Fk8Z2NrO/cV1rNyaz4d7S2m2O+kf6s/aH88mLNCX6sYW5j25GbtT8/cbxnHJ8K96szZkVXDrigzunzecO2a3nx3235vzGNY/hJumDjyt8YvHe0wB5o+J46nFE9hbXMftL2dS2dBCVJCJSUkRWGwO8quaKK5tZlZqNH//9jiWf5HPc5/nMiIulFmp0YyMC8Vk9KG+2c4XOVV8uLeU2FA/fjN/JNeMjUMpxfpD5fzg1S+xOb8qp507Kpa/3TAOg4/iF+/s5cO9pXxjbBx/vX4sgaavyjnrmu08/NEh3t7p7g0bFBNEXqWFH1w8mF/MHcbRKgv/3JDDxsOVmC3utSUDTQbmj4mj2rMUDcDxXDAqyMS4AeEkRwex8XAFuZUWlIIR/UNJS4nk1pnJ7Xrl8qss2J0uEiIC+MU7e/l4fxlvf386vgbFTS/soF+oHz+5fChXjIzF5nSxek8Jz32eS2mtlV9cNYz0ozWszyrnHzd2fsGoyNzE/Cc3U291cMv0JH6/YHTrc8W1zdz0wnYq6ltocTgZFBPMQ9eO4tND5byVUYSf0YfvTk/mprSBVDa0sK+4juToQGYMju7wc97KKOSB9/fjdGkuGxFLQngAb2cW0WRzMiU5gmvHxXPV6DhiQk6ehO0sMLNkeQbRwSZ8DT4U1TSxLMeZtQAAFeVJREFUfMkUUmND2FlQQ3SwicmemXcdThe3rcxka24VP7h4CHdfMrjbSoWzuTSL1prSOisxIX4dLkaU1Vm5+cUd5FQ0Eh3sR2JkAGaLjYJq91jkUy3BPl1Ol+Ynb+1m1Z4SlIK37ug4CdmBkjrufu1LHlowmjltJkPrSZYWByu25hMRaGLaoEhSooP67LjdJpuDAF9Dl79/s81JgOnk1TfffGYL1vO4PBMkwRNCCHGObD5SyQPv7yclOognF0+4YHo/tdZsy6smPiyg26uzDVY7/ztQzsj40HY9g7mVjdz92pdklTWwbGYKyy5KJibEj7mPb8LHR/HJvbPPWsmtw+nixue3Y/BRvLwsrXWMqdli49ND5WzPdS/FERboS2JEIFOSI/jO1KTWHu9Ve0r49+Y8ssoaWnvGwL1G5R2zB3PnnEHtkjRwXzjJKm0g0GQgKtjExIER7XoJn/s8j7+uzWJ4/1Ae+eYYfBTkVVp45ONDVDXaWDojmaUzkkkID+D/PnCXlU5OimBXUS1+Rh+uHh3H1JRI4sL9+XBPKR/uLSHAZODmaUl8Z2pSp4mL1pq9x+rYcLiCjHwzmfk1GH0UD147istHxPKXT7J4M6Oo3fe0La/akVfNT97aTUmdlRB/IzaHixaHi+H9Q/jTwjFMSorAaneyZHk6OwtqGJUQRnyYP2MHhHPztIH4GQ3c8K9t5FU0Mnd0f97ZeYyHrh3ForREtuZU88D7+6m32lm5LA2rzck9b+yi2mLD6KO4dlw89VYHnx4qb9c+k9GH9+6a0bqsj8Pp4o8fHWLF1nxmpUbzl2+NbZ1IqLbJxms7CvlgdzHZ5Y0ARAebGBwTzJiEMKakRBIfFsD6rHI+3lfWWvJb1dhCfHgAb9w+DaNBcdML28mpaKTN5Mk8MH8E35s1iIdWH+ClLfmkpUSSftTM4JggRsaHUV5nxWT04TfzRzAiLhSH08Wjaw+zYms+0cF+JEUFMjQ2hHGJYYxJCCMmxJ9QfyPNdie5FRaO1TQxZkBYp2NRS+uaeWVbAZ/sLyOvyr0UzNM3TWw9LvOrLNz84g5qm+z8e8lkpg36qvexsqGFJcvTqWps4dP75rSef3IqGli1p5SP9paQV2XB1+CDn9GHaYOiuHZcPHFh/ry7q5iP9paiFJ7zQCBTkiOZPjiK6GA/7E4XT64/whvpRdx7WSrv7y7G4dR8/ONZrT/H4XSx4OktHCipJyrIxMf3zqLfScZhH+dyaXIqG8nIN+NyaaKD/XBqzabsSjYfqWJ8Yjh/u2Fcp+Nh26pqbGHZigz2Hqtr3TYiLpSVy6ac8uRa9VY7WnPKZcBaa3YW1HDEU/bfL8SPW6Ynf+31Ztuqa7aTftTMiLiQ0xq7DO6LGbe+lMGw/iH886aJrWPitdZsPlLFMxtzyMiv4eGFo7udHG79oXJuW5l5XpdngiR4QgghzqG+usC61e7k4TWHeHlbAeD+sF3VaOuwrMnZ4HRpfBRn9D7bnS7yKi1oNKH+vkQGmU5rQqITbTxcwY/e2EW91dG6bXj/EP56/VjGDvhqmQ6tNQ+tPsjr6YXcPDWJuy4Z3KFH1uZw4aPoctmQzpTUNvOTt3az46gZf18f7E7NspnJjE4I41hNMwG+BpbMSG5XFuxyuRP7D3YXuxO2yQMYkxDW7n2tt9p57H/Z5FY2UlzbTF6lhfBAX8YkhLk/JH5nInNH9ef7r+zks6xygkxGGlocRAebWHFrWmu5YGldM2v2lTFvTP/WmXhzKhpZd7CcgZGBJEUFctvKDAJ8Day65yKabU7ufXMX2/PM3HZRCr++eniX70dWWT2bs6vIqWgku6KBAyX1rcm7UjAlOZLUfsFY7S78fH2497LU1g+71Y0tPLsxl5gQPyYlRbB8y1HW7CtjVmo0m49UsWxmCr+9ZiSfZ1fy8EeHsDqcxIb6k1dpob7Zzj2XDmFzThXpR83MHxOH0aAoqG7icFkDzXZnaxuNPgqn1u3WZk3tF8z8sXHcOjOFsABf0o+a+cGrO6lttjNtUCRTkiN5aUs+Tpc7lvuK69ia6y6XXrksrd3f1XG7i2pZ+MwWbpmWxG+vGcVj6w7z9IZclIKpKZFMSorA6XInDp9llVNe7x4bG+BrYO6oWIL9jZTWWjlc3tC6/mpbP7xkCD+bO4xdhTVc/9y21omglFI8vymXh9dkcd8VQ3l6Y477/Vw6hVe3F/Kvz3MJDfBlRFwok5MiuG5CAmEBvjRY7Ty7MZc3M4pae7HbCvU3Mjk5ko2HKxgVH8bypVPaXfDQWpNf3YTZ0kKNxc4fPjpIeb2Vfy6eyKCYIL7IqeLPH2cRHx7Am3dMIzrYj7pmO8U1zQyNDe7wN/XB7mLuf3cfNqeLOUP7MX9sf1Kig+kf6k9JXTObsivZX1zHuAHhzB3dn8qGFh5de5jdRbWAuxzf6dIsTkvkT9eNwcdHUVzbzK7CGuLDA0iKDCQyyNTtuavJ5mB7XjUf7C7hk/1lrZNxDY0NZnxiOOGB7nNVQbWFrNIGwgJ9+ceN49vNorwlp4rvrcwkKthEdaONID8Dv7xqODmVjaw/VEFORaNntmZ/viys5TfzRnD77EEd2vJmeiH3v7eP0QlhvLJs6nk9yZ0keEIIIcQ5klPRwKbsKrbkVJEcHcT/fWNkbzepxxTXNpOZbybIZCQ0wJfxieFd9lzaHK6zPvOn06V5YXMemflmfjZ32Ncan3Yy+47V8di6w2w4XMl3pyXxh+vcZZmWFgf3vb2H0AAjV4+OY8aQqNOeeCkj38yi57czOSmCnIpGmmxO/njdaL416fRKzlscTvYdq6OopomZg6NPuRcJ3O/h8fVN5wyN4cUlkztNLM0WG/e/u49PDpQR4GvgkW+O4boJX5WxOpwuciobOVhSj9niXu7Fz2hgaGwwsaH+7CyoYf2hCrblVRMW4Mu8Me5e0MQI98RJxyd/OVbTxI/e2MWXhbUMjAzk0uH9WDIjmZRuetp/98F+Xt5ewITEcL4srOXGyYncN3dohx4sp0uTftRMRYOVy0bEEnxC79ixmiYy8s00tjgxGRT9Qv25eGhMa3LyxKdHePzTbMYkhLE4bSC///AAFw2J4YVbJvFWRhG/encfUUEmqi02pg+KIsjPwKHSBoprmwk0GbhqVH82HamkqtHG1aP7c8nwfkxNiSTAZKC60Ybd6WJkXChGgw/rD5Xzw9d3ERHoy1Wj4xidEEpJbTPv7Som1zPzMkB4oC/Ll05ptzTQ9rxqlr6UzsDIQFKig9iQVYnN6SLEz8jUQVGMjAshLjyAvcfqeCO9kElJEUxIDGf13pLWBPg4pSApMpD86q+W5okP8+eey1KZPTSG/qH+PL4um39uyGHhhAT8fX14Z+cx7M6v8otB0UHMHd2ftJRIKhtaKDI3YbbYaGxxUFZnZVdhLTani1B/I9dNSODKkf3JKqtnw+EKjpQ3Um+1Y7W7iA/zZ3hcKBlHzQSYDCxfOoV+IX78Z+cxnlh/hJSoIF75Xhp1TXbufHUnuZUWjJ6ZpheMj+e6CQkoFD95ezcf7S1lVmo0AyICCAswUddso7jWyqbsSuYMjeGZ70w8ae9pb+u1BE8pdRXwBGAA/q21/vMJz/sBLwOTgGrgRq11fnevKQmeEEIIIXpakbmJhPCAs1KG1ta/Ps/lkY+zGBYbwtPfmcCQfj0/Dvj4hEXjEsM6lOueuN9nWRUkRwcx+GvMWAzu8a1//587YZ49NIanFk3o0EvidGmqGlvoF+J3Sr3W9VY7Vzz2OTUWOw8tGMXiczQTqsul+c/OIp7dmEt+dRPBfkbW/XQ2cWEBaK35+Tt72VVYw6+vHsFlI/q1tn1/cR0rtuazak8J4weE85v5IxiX2LE38kS7i2r500cH2Vdc17o2blpKJNeMi2dgZCDBfkYGxwQRHthxNt0tOVXctjKDEH9fvjE2jtHxYWQWmNmWW02huam1RPf7cwbxsyuH4WvwwenSZJXVU1JrpazeSniALxcNiSYiyER5vZV1B8sx+igWTkzocDHjePJrMviwKC2Rb00cQFVjC0erLHyeXcm23Gocnh/qo9zr6Ab7GwkP8CUtJZLZQ2OYkhzZZVWBw+lqvfBwuKyBZSsyqGpsweHSOF2a2UNjeOLG8a1rlFpaHOwrrmNUfGiHMc9Ol+bRtYfZeLiCqkYbdc02wgJMRAebmDkkml9dPbzTSanON72S4CmlDEA2cAVwDMgAFmutD7bZ5y5grNb6TqXUImCh1vrG7l5XEjwhhBBCeAutNdtyq5kwMOKUJn/wFkXmJuLDA046s+6pKqxuwu5yfe3E83Q4nC7WHSwnKtivw6Qr3fm6y304nC7yqiwE+xnblSWeTF2znWC/jjMmO5wuyhtacLk0iZGnN86tO1tzqhgUE9xhplpwjyHNKmsgLsyf+PCAM06gKuqt/PaDA6TEBPHtyYnd9vB6q95K8KYDD2qt53oe/xpAa/1Im33WevbZppQyAmVAjO6mUZLgCSGEEEIIIfqy7hK8c9n/mAC0nc7qmGdbp/torR1AHdBhcRal1B1KqUylVGZlZeU5aq4QQgghhBBCXNjO/wJTQGv9vNZ6stZ6ckxM76wxIoQQQgghhBDnu3OZ4BUDiW0eD/Bs63QfT4lmGO7JVoQQQgghhBBCnKZzmeBlAKlKqRSllAlYBKw6YZ9VwBLP/euBz7obfyeEEEIIIYQQomvnbIEHrbVDKfVDYC3uZRKWa60PKKV+D2RqrVcBLwKvKKVyADPuJFAIIYQQQgghxNdwTlfw01qvAdacsO23be5bgRvOZRuEEEIIIYQQoq+4ICZZEUIIIYQQQghxcpLgCSGEEEIIIYSXkARPCCGEEEIIIbyEJHhCCCGEEEII4SUkwRNCCCGEEEIILyEJnhBCCCGEEEJ4CUnwhBBCCCGEEMJLSIInhBBCCCGEEF5CEjwhhBBCCCGE8BKS4AkhhBBCCCGEl5AETwghhBBCCCG8hCR4QgghhBBCCOEllNa6t9twWpRSlUBBb7ejE9FAVW83QvQKiX3fJvHvuyT2fZfEvu+S2Pdt51P8k7TWMZ09ccEleOcrpVSm1npyb7dD9DyJfd8m8e+7JPZ9l8S+75LY920XSvylRFMIIYQQQgghvIQkeEIIIYQQQgjhJSTBO3ue7+0GiF4jse/bJP59l8S+75LY910S+77tgoi/jMETQgghhBBCCC8hPXhCCCGEEEII4SUkwTsLlFJXKaUOK6VylFK/6u32iHNLKZWvlNqnlNqtlMr0bItUSq1TSh3x3Eb0djvFmVNKLVdKVSil9rfZ1mmslduTnvPAXqXUxN5ruTgbuoj/g0qpYs/xv1spNa/Nc7/2xP+wUmpu77RanA1KqUSl1Aal1EGl1AGl1L2e7XL8e7luYi/HvpdTSvkrpdKVUns8sX/Isz1FKbXDE+O3lFImz3Y/z+Mcz/PJvdn+tiTBO0NKKQPwNHA1MBJYrJQa2butEj3gEq31+DZT5f4KWK+1TgXWex6LC98K4KoTtnUV66uBVM/XHcCzPdRGce6soGP8AR73HP/jtdZrADzn/UXAKM/3POP5/yAuTA7gPq31SGAacLcnxnL8e7+uYg9y7Hu7FuBSrfU4YDxwlVJqGvAX3LEfAtQAt3n2vw2o8Wx/3LPfeUESvDOXBuRorfO01jbgTWBBL7dJ9LwFwErP/ZXAdb3YFnGWaK03AeYTNncV6wXAy9ptOxCulIrrmZaKc6GL+HdlAfCm1rpFa30UyMH9/0FcgLTWpVrrLz33G4BDQAJy/Hu9bmLfFTn2vYTn+G30PPT1fGngUuAdz/YTj/vj54N3gMuUUqqHmtstSfDOXAJQ1ObxMbo/EYgLnwb+p5TaqZS6w7MtVmtd6rlfBsT2TtNED+gq1nIu6Dt+6CnDW96mHFvi76U8ZVcTgB3I8d+nnBB7kGPf6ymlDEqp3UAFsA7IBWq11g7PLm3j2xp7z/N1QFTPtrhzkuAJcfou0lpPxF2Sc7dSanbbJ7V7alqZnrYPkFj3Sc8Cg3GX75QCf+/d5ohzSSkVDPwX+LHWur7tc3L8e7dOYi/Hfh+gtXZqrccDA3D3xA7v5SZ9LZLgnbliILHN4wGebcJLaa2LPbcVwHu4TwDlx8txPLcVvddCcY51FWs5F/QBWutyzwcAF/ACX5ViSfy9jFLKF/cH/Ne01u96Nsvx3wd0Fns59vsWrXUtsAGYjrvk2uh5qm18W2PveT4MqO7hpnZKErwzlwGkembYMeEeaLuql9skzhGlVJBSKuT4feBKYD/umC/x7LYE+KB3Wih6QFexXgXc4plNbxpQ16aUS3iJE8ZVLcR9/IM7/os8s6ql4J5sI72n2yfODs84mheBQ1rrx9o8Jce/l+sq9nLsez+lVIxSKtxzPwC4AvcYzA3A9Z7dTjzuj58Prgc+0+fJAuPGk+8iuqO1diilfgisBQzAcq31gV5uljh3YoH3PGNojcDrWutPlFIZwNtKqduAAuDbvdhGcZYopd4ALgailVLHgN8Bf6bzWK8B5uEeYN8E3NrjDRZnVRfxv1gpNR53aV4+8H0ArfUBpdTbwEHcs/DdrbV29ka7xVkxE/gusM8zHgfgfuT47wu6iv1iOfa9Xhyw0jMLqg/wttb6Q6XUQeBNpdQfgV24LwDguX1FKZWDe0KuRb3R6M6o8yTRFEIIIYQQQghxhqREUwghhBBCCCG8hCR4QgghhBBCCOElJMETQgghhBBCCC8hCZ4QQgghhBBCeAlJ8IQQQgghhBDCS0iCJ4QQos9SSjmVUruVUnuUUl8qpWacZP9wpdRdp/C6G5VSk89eS4UQQohTIwmeEEKIvqxZaz1eaz0O+DXwyEn2DwdOmuAJIYQQvUUSPCGEEMItFKgBUEoFK6XWe3r19imlFnj2+TMw2NPr96hn31969tmjlPpzm9e7QSmVrpTKVkrN6tlfRQghRF9l7O0GCCGEEL0oQCm1G/AH4oBLPdutwEKtdb1SKhrYrpRaBfwKGK21Hg+glLoaWABM1Vo3KaUi27y2UWudppSaB/wOuLyHfichhBB9mCR4Qggh+rLmNsnadOBlpdRoQAEPK6VmAy4gAYjt5PsvB17SWjcBaK3NbZ5713O7E0g+N80XQggh2pMETwghhAC01ts8vXUxwDzP7SSttV0plY+7l+90tHhuncj/WyGEED1ExuAJIYQQgFJqOGAAqoEwoMKT3F0CJHl2awBC2nzbOuBWpVSg5zXalmgKIYQQPU6uKAohhOjLjo/BA3dZ5hKttVMp9RqwWim1D8gEsgC01tVKqS1Kqf3Ax1rrnyulxgOZSikbsAa4vxd+DyGEEAIApbXu7TYIIYQQQgghhDgLpERTCCGEEEIIIbyEJHhCCCGEEEII4SUkwRNCCCGEEEIILyEJnhBCCCGEEEJ4CUnwhBBCCCGEEMJLSIInhBBCCCGEEF5CEjwhhBBCCCGE8BKS4AkhhBBCCCGEl/h/FhG4iLTjYvgAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] @@ -1055,24 +1359,13 @@ "plt.plot(train_loss_set)\n", "plt.show()" ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "udNKTdbVGcxZ" - }, - "outputs": [], - "source": [] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], - "name": "BERT_SNIPS.ipynb", + "name": "05_BERT_SNIPS.ipynb", "provenance": [] }, "kernelspec": { @@ -1090,7 +1383,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.10" + "version": "3.7.4" } }, "nbformat": 4, diff --git a/Ch6/Data/data/atis.test.pkl b/Ch6/Data/data/atis.test.pkl new file mode 100644 index 0000000..4988a4a Binary files /dev/null and b/Ch6/Data/data/atis.test.pkl differ diff --git a/Ch6/Data/data/atis.train.pkl b/Ch6/Data/data/atis.train.pkl new file mode 100644 index 0000000..e871452 Binary files /dev/null and b/Ch6/Data/data/atis.train.pkl differ