examples/notebooks/03-active-learning-with-setfit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bea8b7ed",
   "metadata": {},
   "source": [
    "# Use SetFit for Active Learning in Small-Text\n",
    "*Notebook 3*  \n",
    "In this tutorial we demonstrate how to use [SetFit](https://github.com/huggingface/setfit) as a classifier for active learning.\n",
    "\n",
    "----\n",
    "\n",
    "## Overview\n",
    "\n",
    "- [Part I: Installation](#nb3-part1-installation)\n",
    "- [Part II: Data](#nb3-part2-data)\n",
    "- [Part III: Active Learning](#nb3-part3-active-learning)\n",
    "- [Part IV: Comparison](#nb3-part4-comparison)\n",
    "\n",
    "----\n",
    "\n",
    "<a id=\"nb3-part1-installation\"></a>\n",
    "## I. Installation\n",
    "\n",
    "The small-text wrapper for SetFit is a part of the transformers integration, so the installation procedure is the same as for every other example that shows small-text with transformers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "837556c3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting small-text[transformers]\n",
      "  Cloning https://github.com/webis-de/small-text.git (to revision dev) to /tmp/pip-install-j5wlnpex/small-text_1830d7022f7a4b6290cfa34732379ebe\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/webis-de/small-text.git /tmp/pip-install-j5wlnpex/small-text_1830d7022f7a4b6290cfa34732379ebe\n",
      "  Running command git checkout -b dev --track origin/dev\n",
      "  Switched to a new branch 'dev'\n",
      "  Branch 'dev' set up to track remote branch 'dev' from 'origin'.\n",
      "  Resolved https://github.com/webis-de/small-text.git to commit 5edee70c64dbb2ecf103dc8775f62b1339fa8c9e\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: packaging in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from small-text[transformers]) (23.0)\n",
      "Requirement already satisfied: dill in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from small-text[transformers]) (0.3.6)\n",
      "Requirement already satisfied: scikit-learn>=0.24.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from small-text[transformers]) (1.2.1)\n",
      "Requirement already satisfied: numpy>=1.20.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from small-text[transformers]) (1.24.1)\n",
      "Requirement already satisfied: scipy in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from small-text[transformers]) (1.10.0)\n",
      "Requirement already satisfied: tqdm in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from small-text[transformers]) (4.64.1)\n",
      "Requirement already satisfied: torchtext>=0.7.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from small-text[transformers]) (0.14.1)\n",
      "Requirement already satisfied: torch>=1.6.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from small-text[transformers]) (1.13.1)\n",
      "Requirement already satisfied: transformers>=4.0.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from small-text[transformers]) (4.26.0)\n",
      "Requirement already satisfied: joblib>=1.1.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from scikit-learn>=0.24.1->small-text[transformers]) (1.2.0)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from scikit-learn>=0.24.1->small-text[transformers]) (3.1.0)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from torch>=1.6.0->small-text[transformers]) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from torch>=1.6.0->small-text[transformers]) (11.10.3.66)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from torch>=1.6.0->small-text[transformers]) (8.5.0.96)\n",
      "Requirement already satisfied: typing-extensions in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from torch>=1.6.0->small-text[transformers]) (4.4.0)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from torch>=1.6.0->small-text[transformers]) (11.7.99)\n",
      "Requirement already satisfied: wheel in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.6.0->small-text[transformers]) (0.38.4)\n",
      "Requirement already satisfied: setuptools in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.6.0->small-text[transformers]) (66.1.1)\n",
      "Requirement already satisfied: requests in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from torchtext>=0.7.0->small-text[transformers]) (2.28.2)\n",
      "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from transformers>=4.0.0->small-text[transformers]) (0.13.2)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from transformers>=4.0.0->small-text[transformers]) (2022.10.31)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from transformers>=4.0.0->small-text[transformers]) (0.12.0)\n",
      "Requirement already satisfied: filelock in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from transformers>=4.0.0->small-text[transformers]) (3.9.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from transformers>=4.0.0->small-text[transformers]) (6.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from requests->torchtext>=0.7.0->small-text[transformers]) (2.1.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from requests->torchtext>=0.7.0->small-text[transformers]) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from requests->torchtext>=0.7.0->small-text[transformers]) (1.26.14)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from requests->torchtext>=0.7.0->small-text[transformers]) (2022.12.7)\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Collecting sentence-transformers\n",
      "  Cloning https://github.com/chschroeder/sentence-transformers to /tmp/pip-install-no5qrler/sentence-transformers_739ea867a2a2421f993b3ecd0af43083\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/chschroeder/sentence-transformers /tmp/pip-install-no5qrler/sentence-transformers_739ea867a2a2421f993b3ecd0af43083\n",
      "  Resolved https://github.com/chschroeder/sentence-transformers to commit f54ba243db9fd614de597c4ce96833690d4abaa4\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting pip\n",
      "  Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)\n",
      "Collecting install\n",
      "  Using cached install-1.3.5-py3-none-any.whl (3.2 kB)\n",
      "Collecting transformers<5.0.0,>=4.6.0\n",
      "  Using cached transformers-4.26.0-py3-none-any.whl (6.3 MB)\n",
      "Collecting tqdm\n",
      "  Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)\n",
      "Collecting torch>=1.6.0\n",
      "  Using cached torch-1.13.1-cp38-cp38-manylinux1_x86_64.whl (887.4 MB)\n",
      "Collecting torchvision\n",
      "  Using cached torchvision-0.14.1-cp38-cp38-manylinux1_x86_64.whl (24.2 MB)\n",
      "Collecting numpy\n",
      "  Using cached numpy-1.24.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n",
      "Collecting scikit-learn\n",
      "  Using cached scikit_learn-1.2.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.8 MB)\n",
      "Collecting scipy\n",
      "  Using cached scipy-1.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)\n",
      "Collecting nltk\n",
      "  Using cached nltk-3.8.1-py3-none-any.whl (1.5 MB)\n",
      "Collecting sentencepiece\n",
      "  Using cached sentencepiece-0.1.97-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "Collecting huggingface-hub>=0.4.0\n",
      "  Using cached huggingface_hub-0.12.0-py3-none-any.whl (190 kB)\n",
      "Collecting pyyaml>=5.1\n",
      "  Using cached PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)\n",
      "Collecting filelock\n",
      "  Using cached filelock-3.9.0-py3-none-any.whl (9.7 kB)\n",
      "Collecting typing-extensions>=3.7.4.3\n",
      "  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)\n",
      "Collecting requests\n",
      "  Using cached requests-2.28.2-py3-none-any.whl (62 kB)\n",
      "Collecting packaging>=20.9\n",
      "  Using cached packaging-23.0-py3-none-any.whl (42 kB)\n",
      "Collecting nvidia-cuda-runtime-cu11==11.7.99\n",
      "  Using cached nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n",
      "Collecting nvidia-cudnn-cu11==8.5.0.96\n",
      "  Using cached nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n",
      "Collecting nvidia-cuda-nvrtc-cu11==11.7.99\n",
      "  Using cached nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)\n",
      "Collecting nvidia-cublas-cu11==11.10.3.66\n",
      "  Using cached nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n",
      "Collecting setuptools\n",
      "  Using cached setuptools-66.1.1-py3-none-any.whl (1.3 MB)\n",
      "Collecting wheel\n",
      "  Using cached wheel-0.38.4-py3-none-any.whl (36 kB)\n",
      "Collecting regex!=2019.12.17\n",
      "  Using cached regex-2022.10.31-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (772 kB)\n",
      "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
      "  Using cached tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
      "Collecting click\n",
      "  Using cached click-8.1.3-py3-none-any.whl (96 kB)\n",
      "Collecting joblib\n",
      "  Using cached joblib-1.2.0-py3-none-any.whl (297 kB)\n",
      "Collecting threadpoolctl>=2.0.0\n",
      "  Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)\n",
      "Collecting pillow!=8.3.*,>=5.3.0\n",
      "  Using cached Pillow-9.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)\n",
      "Collecting charset-normalizer<4,>=2\n",
      "  Using cached charset_normalizer-3.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (195 kB)\n",
      "Collecting urllib3<1.27,>=1.21.1\n",
      "  Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB)\n",
      "Collecting certifi>=2017.4.17\n",
      "  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)\n",
      "Collecting idna<4,>=2.5\n",
      "  Using cached idna-3.4-py3-none-any.whl (61 kB)\n",
      "Building wheels for collected packages: sentence-transformers\n",
      "  Building wheel for sentence-transformers (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=126137 sha256=12228d93dd42691db52ee075edffc1a8b2d485b854e863cfa5055cc4f722e53b\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-wbsz70xp/wheels/f9/e6/0c/2a4c6a5dcbc9ba59b9dd63c5042a6753c4506dbb1f4b0b31ec\n",
      "Successfully built sentence-transformers\n",
      "Installing collected packages: tokenizers, sentencepiece, charset-normalizer, wheel, urllib3, typing-extensions, tqdm, threadpoolctl, setuptools, regex, pyyaml, pip, pillow, packaging, nvidia-cuda-nvrtc-cu11, numpy, joblib, install, idna, filelock, click, certifi, scipy, requests, nvidia-cuda-runtime-cu11, nvidia-cublas-cu11, nltk, scikit-learn, nvidia-cudnn-cu11, huggingface-hub, transformers, torch, torchvision, sentence-transformers\n",
      "  Attempting uninstall: tokenizers\n",
      "    Found existing installation: tokenizers 0.13.2\n",
      "    Uninstalling tokenizers-0.13.2:\n",
      "      Successfully uninstalled tokenizers-0.13.2\n",
      "  Attempting uninstall: sentencepiece\n",
      "    Found existing installation: sentencepiece 0.1.97\n",
      "    Uninstalling sentencepiece-0.1.97:\n",
      "      Successfully uninstalled sentencepiece-0.1.97\n",
      "  Attempting uninstall: charset-normalizer\n",
      "    Found existing installation: charset-normalizer 2.1.1\n",
      "    Uninstalling charset-normalizer-2.1.1:\n",
      "      Successfully uninstalled charset-normalizer-2.1.1\n",
      "  Attempting uninstall: wheel\n",
      "    Found existing installation: wheel 0.38.4\n",
      "    Uninstalling wheel-0.38.4:\n",
      "      Successfully uninstalled wheel-0.38.4\n",
      "  Attempting uninstall: urllib3\n",
      "    Found existing installation: urllib3 1.26.14\n",
      "    Uninstalling urllib3-1.26.14:\n",
      "      Successfully uninstalled urllib3-1.26.14\n",
      "  Attempting uninstall: typing-extensions\n",
      "    Found existing installation: typing_extensions 4.4.0\n",
      "    Uninstalling typing_extensions-4.4.0:\n",
      "      Successfully uninstalled typing_extensions-4.4.0\n",
      "  Attempting uninstall: tqdm\n",
      "    Found existing installation: tqdm 4.64.1\n",
      "    Uninstalling tqdm-4.64.1:\n",
      "      Successfully uninstalled tqdm-4.64.1\n",
      "  Attempting uninstall: threadpoolctl\n",
      "    Found existing installation: threadpoolctl 3.1.0\n",
      "    Uninstalling threadpoolctl-3.1.0:\n",
      "      Successfully uninstalled threadpoolctl-3.1.0\n",
      "  Attempting uninstall: setuptools\n",
      "    Found existing installation: setuptools 66.1.1\n",
      "    Uninstalling setuptools-66.1.1:\n",
      "      Successfully uninstalled setuptools-66.1.1\n",
      "  Attempting uninstall: regex\n",
      "    Found existing installation: regex 2022.10.31\n",
      "    Uninstalling regex-2022.10.31:\n",
      "      Successfully uninstalled regex-2022.10.31\n",
      "  Attempting uninstall: pyyaml\n",
      "    Found existing installation: PyYAML 6.0\n",
      "    Uninstalling PyYAML-6.0:\n",
      "      Successfully uninstalled PyYAML-6.0\n",
      "  Attempting uninstall: pip\n",
      "    Found existing installation: pip 22.3.1\n",
      "    Uninstalling pip-22.3.1:\n",
      "      Successfully uninstalled pip-22.3.1\n",
      "  Attempting uninstall: pillow\n",
      "    Found existing installation: Pillow 9.4.0\n",
      "    Uninstalling Pillow-9.4.0:\n",
      "      Successfully uninstalled Pillow-9.4.0\n",
      "  Attempting uninstall: packaging\n",
      "    Found existing installation: packaging 23.0\n",
      "    Uninstalling packaging-23.0:\n",
      "      Successfully uninstalled packaging-23.0\n",
      "  Attempting uninstall: nvidia-cuda-nvrtc-cu11\n",
      "    Found existing installation: nvidia-cuda-nvrtc-cu11 11.7.99\n",
      "    Uninstalling nvidia-cuda-nvrtc-cu11-11.7.99:\n",
      "      Successfully uninstalled nvidia-cuda-nvrtc-cu11-11.7.99\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 1.24.1\n",
      "    Uninstalling numpy-1.24.1:\n",
      "      Successfully uninstalled numpy-1.24.1\n",
      "  Attempting uninstall: joblib\n",
      "    Found existing installation: joblib 1.2.0\n",
      "    Uninstalling joblib-1.2.0:\n",
      "      Successfully uninstalled joblib-1.2.0\n",
      "  Attempting uninstall: install\n",
      "    Found existing installation: install 1.3.5\n",
      "    Uninstalling install-1.3.5:\n",
      "      Successfully uninstalled install-1.3.5\n",
      "  Attempting uninstall: idna\n",
      "    Found existing installation: idna 3.4\n",
      "    Uninstalling idna-3.4:\n",
      "      Successfully uninstalled idna-3.4\n",
      "  Attempting uninstall: filelock\n",
      "    Found existing installation: filelock 3.9.0\n",
      "    Uninstalling filelock-3.9.0:\n",
      "      Successfully uninstalled filelock-3.9.0\n",
      "  Attempting uninstall: click\n",
      "    Found existing installation: click 8.1.3\n",
      "    Uninstalling click-8.1.3:\n",
      "      Successfully uninstalled click-8.1.3\n",
      "  Attempting uninstall: certifi\n",
      "    Found existing installation: certifi 2022.12.7\n",
      "    Uninstalling certifi-2022.12.7:\n",
      "      Successfully uninstalled certifi-2022.12.7\n",
      "  Attempting uninstall: scipy\n",
      "    Found existing installation: scipy 1.10.0\n",
      "    Uninstalling scipy-1.10.0:\n",
      "      Successfully uninstalled scipy-1.10.0\n",
      "  Attempting uninstall: requests\n",
      "    Found existing installation: requests 2.28.2\n",
      "    Uninstalling requests-2.28.2:\n",
      "      Successfully uninstalled requests-2.28.2\n",
      "  Attempting uninstall: nvidia-cuda-runtime-cu11\n",
      "    Found existing installation: nvidia-cuda-runtime-cu11 11.7.99\n",
      "    Uninstalling nvidia-cuda-runtime-cu11-11.7.99:\n",
      "      Successfully uninstalled nvidia-cuda-runtime-cu11-11.7.99\n",
      "  Attempting uninstall: nvidia-cublas-cu11\n",
      "    Found existing installation: nvidia-cublas-cu11 11.10.3.66\n",
      "    Uninstalling nvidia-cublas-cu11-11.10.3.66:\n",
      "      Successfully uninstalled nvidia-cublas-cu11-11.10.3.66\n",
      "  Attempting uninstall: nltk\n",
      "    Found existing installation: nltk 3.8.1\n",
      "    Uninstalling nltk-3.8.1:\n",
      "      Successfully uninstalled nltk-3.8.1\n",
      "  Attempting uninstall: scikit-learn\n",
      "    Found existing installation: scikit-learn 1.2.1\n",
      "    Uninstalling scikit-learn-1.2.1:\n",
      "      Successfully uninstalled scikit-learn-1.2.1\n",
      "  Attempting uninstall: nvidia-cudnn-cu11\n",
      "    Found existing installation: nvidia-cudnn-cu11 8.5.0.96\n",
      "    Uninstalling nvidia-cudnn-cu11-8.5.0.96:\n",
      "      Successfully uninstalled nvidia-cudnn-cu11-8.5.0.96\n",
      "  Attempting uninstall: huggingface-hub\n",
      "    Found existing installation: huggingface-hub 0.12.0\n",
      "    Uninstalling huggingface-hub-0.12.0:\n",
      "      Successfully uninstalled huggingface-hub-0.12.0\n",
      "  Attempting uninstall: transformers\n",
      "    Found existing installation: transformers 4.26.0\n",
      "    Uninstalling transformers-4.26.0:\n",
      "      Successfully uninstalled transformers-4.26.0\n",
      "  Attempting uninstall: torch\n",
      "    Found existing installation: torch 1.13.1\n",
      "    Uninstalling torch-1.13.1:\n",
      "      Successfully uninstalled torch-1.13.1\n",
      "  Attempting uninstall: torchvision\n",
      "    Found existing installation: torchvision 0.14.1\n",
      "    Uninstalling torchvision-0.14.1:\n",
      "      Successfully uninstalled torchvision-0.14.1\n",
      "  Attempting uninstall: sentence-transformers\n",
      "    Found existing installation: sentence-transformers 2.2.2\n",
      "    Uninstalling sentence-transformers-2.2.2:\n",
      "      Successfully uninstalled sentence-transformers-2.2.2\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "aiohttp 3.8.3 requires charset-normalizer<3.0,>=2.0, but you have charset-normalizer 3.0.1 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed certifi-2022.12.7 charset-normalizer-3.0.1 click-8.1.3 filelock-3.9.0 huggingface-hub-0.12.0 idna-3.4 install-1.3.5 joblib-1.2.0 nltk-3.8.1 numpy-1.24.1 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 packaging-23.0 pillow-9.4.0 pip-22.3.1 pyyaml-6.0 regex-2022.10.31 requests-2.28.2 scikit-learn-1.2.1 scipy-1.10.0 sentence-transformers-2.2.2 sentencepiece-0.1.97 setuptools-66.1.1 threadpoolctl-3.1.0 tokenizers-0.13.2 torch-1.13.1 torchvision-0.14.1 tqdm-4.64.1 transformers-4.26.0 typing-extensions-4.4.0 urllib3-1.26.14 wheel-0.38.4\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Requirement already satisfied: datasets in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (2.7.1)\n",
      "Requirement already satisfied: numpy>=1.17 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (1.24.1)\n",
      "Requirement already satisfied: tqdm>=4.62.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (4.64.1)\n",
      "Requirement already satisfied: xxhash in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (3.1.0)\n",
      "Requirement already satisfied: pandas in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (1.5.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (6.0)\n",
      "Requirement already satisfied: pyarrow>=6.0.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (10.0.1)\n",
      "Requirement already satisfied: responses<0.19 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (0.18.0)\n",
      "Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (0.12.0)\n",
      "Requirement already satisfied: packaging in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (23.0)\n",
      "Requirement already satisfied: fsspec[http]>=2021.11.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (2022.11.0)\n",
      "Requirement already satisfied: aiohttp in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (3.8.3)\n",
      "Requirement already satisfied: requests>=2.19.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (2.28.2)\n",
      "Requirement already satisfied: dill<0.3.7 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (0.3.6)\n",
      "Requirement already satisfied: multiprocess in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from datasets) (0.70.14)\n",
      "Collecting charset-normalizer<3.0,>=2.0\n",
      "  Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from aiohttp->datasets) (1.8.2)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from aiohttp->datasets) (1.3.1)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from aiohttp->datasets) (4.0.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from aiohttp->datasets) (1.3.3)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from aiohttp->datasets) (22.1.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from aiohttp->datasets) (6.0.3)\n",
      "Requirement already satisfied: filelock in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (3.9.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (4.4.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (3.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (2022.12.7)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from requests>=2.19.0->datasets) (1.26.14)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from pandas->datasets) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from pandas->datasets) (2022.6)\n",
      "Requirement already satisfied: six>=1.5 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)\n",
      "Installing collected packages: charset-normalizer\n",
      "  Attempting uninstall: charset-normalizer\n",
      "    Found existing installation: charset-normalizer 3.0.1\n",
      "    Uninstalling charset-normalizer-3.0.1:\n",
      "      Successfully uninstalled charset-normalizer-3.0.1\n",
      "Successfully installed charset-normalizer-2.1.1\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Requirement already satisfied: matplotlib in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (3.4.3)\n",
      "Requirement already satisfied: pillow>=6.2.0 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from matplotlib) (9.4.0)\n",
      "Requirement already satisfied: pyparsing>=2.2.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from matplotlib) (3.0.9)\n",
      "Requirement already satisfied: numpy>=1.16 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from matplotlib) (1.24.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from matplotlib) (0.11.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from matplotlib) (1.3.2)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from matplotlib) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in /home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# until the next release: install small-text from dev branch \n",
    "%pip install git+https://github.com/webis-de/small-text.git@dev#egg=small-text[transformers]\n",
    "# setfit is an optional dependency and must be installed as well\n",
    "%pip install setfit>=0.5.0\n",
    "# temp fix until the gpu memory issue is resolved: https://github.com/UKPLab/sentence-transformers/issues/1793\n",
    "%pip install pip install \"git+https://github.com/chschroeder/sentence-transformers#egg=sentence-transformers\" --force-reinstall\n",
    "%pip install datasets\n",
    "%pip install matplotlib\n",
    "\n",
    "import logging\n",
    "\n",
    "import datasets\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rc('figure', titlesize=22)\n",
    "plt.rc('axes', titlesize=22, labelsize=20, linewidth=1.2)\n",
    "plt.rc('xtick', labelsize=14)\n",
    "plt.rc('ytick', labelsize=14)\n",
    "plt.rc('legend', fontsize=16)\n",
    "plt.rc('lines', linewidth=2)\n",
    "\n",
    "datasets.logging.set_verbosity_error()\n",
    "\n",
    "# disables the progress bar for notebooks: https://github.com/huggingface/datasets/issues/2651\n",
    "datasets.logging.get_verbosity = lambda: logging.NOTSET\n",
    "\n",
    "for logger_name in ['setfit.modeling', 'setfit.trainer']:\n",
    "    logger = logging.getLogger(logger_name)\n",
    "    logger.setLevel(logging.ERROR)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf5fd19",
   "metadata": {},
   "source": [
    "<a id=\"nb3-part2-data\"></a>\n",
    "## II. Data\n",
    "\n",
    "We use the \"AG's News\" dataset for this demonstration. It consists of news articles which are categorized into four mutually exclusive classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "754e5b56",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1122.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 10 training samples:\n",
      "\n",
      "2   Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
      "2   Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
      "2   Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
      "2   Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
      "2   Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n",
      "2   Stocks End Up, But Near Year Lows (Reuters) Reuters - Stocks ended slightly higher on Friday\\but stayed near lows for the year as oil prices surged past  #36;46\\a barrel, offsetting a positive outlook from computer maker\\Dell Inc. (DELL.O)\n",
      "2   Money Funds Fell in Latest Week (AP) AP - Assets of the nation's retail money market mutual funds fell by  #36;1.17 billion in the latest week to  #36;849.98 trillion, the Investment Company Institute said Thursday.\n",
      "2   Fed minutes show dissent over inflation (USATODAY.com) USATODAY.com - Retail sales bounced back a bit in July, and new claims for jobless benefits fell last week, the government said Thursday, indicating the economy is improving from a midsummer slump.\n",
      "2   Safety Net (Forbes.com) Forbes.com - After earning a PH.D. in Sociology, Danny Bazil Riley started to work as the general manager at a commercial real estate firm at an annual base salary of  #36;70,000. Soon after, a financial planner stopped by his desk to drop off brochures about insurance benefits available through his employer. But, at 32, \"buying insurance was the furthest thing from my mind,\" says Riley.\n",
      "2   Wall St. Bears Claw Back Into the Black  NEW YORK (Reuters) - Short-sellers, Wall Street's dwindling  band of ultra-cynics, are seeing green again.\n"
     ]
    }
   ],
   "source": [
    "import datasets\n",
    "import logging\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "# disables the progress bar for notebooks: https://github.com/huggingface/datasets/issues/2651\n",
    "datasets.logging.get_verbosity = lambda: logging.NOTSET\n",
    "\n",
    "raw_dataset = datasets.load_dataset('ag_news')\n",
    "num_classes = np.unique(raw_dataset['train']['label']).shape[0]\n",
    "\n",
    "print('First 10 training samples:\\n')\n",
    "for i in range(10):\n",
    "    print(raw_dataset['train']['label'][i], ' ', raw_dataset['train']['text'][i])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1742f7d7",
   "metadata": {},
   "source": [
    "### Preparing the Data\n",
    "\n",
    "The SetFit Classifier uses the `TextData` structure as input. Such datasets can be easily created from raw text and labels using `TextDataset.from_arrays()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "be151ce8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/utils/annotations.py:67: ExperimentalWarning: The function from_arrays is experimental and maybe subject to change soon.\n",
      "  warnings.warn(f'The {subject} {func_or_class.__name__} is experimental '\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from small_text import TextDataset\n",
    "\n",
    "num_classes = raw_dataset['train'].features['label'].num_classes\n",
    "\n",
    "target_labels = np.arange(num_classes)\n",
    "\n",
    "train = TextDataset.from_arrays(raw_dataset['train']['text'], np.array(raw_dataset['train']['label']), target_labels=target_labels)\n",
    "test = TextDataset.from_arrays(raw_dataset['test']['text'], np.array(raw_dataset['test']['label']), target_labels=target_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3db26811",
   "metadata": {},
   "source": [
    "<a id=\"nb3-part3-active-learning\"></a>\n",
    "## III. Active Learning \n",
    "\n",
    "### Configuring the Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2184e4b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from small_text.integrations.transformers.classifiers.setfit import SetFitModelArguments\n",
    "from small_text.integrations.transformers.classifiers.factories import SetFitClassificationFactory\n",
    "\n",
    "\n",
    "num_classes = 4\n",
    "\n",
    "sentence_transformer_model_name = 'sentence-transformers/paraphrase-mpnet-base-v2'\n",
    "setfit_model_args = SetFitModelArguments(sentence_transformer_model_name)\n",
    "clf_factory = SetFitClassificationFactory(setfit_model_args, \n",
    "                                          num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "017ebf3d",
   "metadata": {},
   "source": [
    "### Setting up the Active Learner\n",
    "\n",
    "Here, we construct a `PoolBasedActiveLearner` instance which requires a classifier factory, a query strategy, and the train dataset.\n",
    "\n",
    "To obtain a first model, we initialize the active learner by providing the true labels for 20 sentences. This corresponds to an initial labeling the real-world setting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "66774521",
   "metadata": {},
   "outputs": [],
   "source": [
    "from small_text import (\n",
    "    PoolBasedActiveLearner, \n",
    "    random_initialization_balanced,\n",
    "    BreakingTies,\n",
    "    SubsamplingQueryStrategy\n",
    ")\n",
    "\n",
    "# define a query strategy and initialize a pool-based active learner\n",
    "query_strategy = SubsamplingQueryStrategy(BreakingTies())\n",
    "# suppress progress bars in jupyter notebook\n",
    "setfit_train_kwargs = {'show_progress_bar': False}\n",
    "active_learner = PoolBasedActiveLearner(clf_factory, query_strategy, train, fit_kwargs={'setfit_train_kwargs': setfit_train_kwargs})\n",
    "\n",
    "\n",
    "# simulate a warm start\n",
    "def initialize_active_learner(active_learner, y_train):\n",
    "\n",
    "    x_indices_initial = random_initialization_balanced(y_train, n_samples=20)\n",
    "    y_initial = y_train[x_indices_initial]\n",
    "\n",
    "    active_learner.initialize_data(x_indices_initial, y_initial)\n",
    "\n",
    "    return x_indices_initial\n",
    "\n",
    "\n",
    "initial_indices = initialize_active_learner(active_learner, train.y)\n",
    "labeled_indices = initial_indices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8888591",
   "metadata": {},
   "source": [
    "### Active Learning Loop\n",
    "\n",
    "The main active learning loop queries the unlabeled pool and thereby decides which documents are labeled next.\n",
    "We then provide the labels for those documents and the active learner retrains the model.\n",
    "After each query, we evaluate the current model against the test set and save the result.\n",
    "\n",
    "\n",
    "Note: This is active learning as it is done in a scientific simulation. In reality, the label feedback would have been given by human annotators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "aacd866c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 1.00\n",
      "Test accuracy: 0.82\n",
      "---------------\n",
      "Iteration #0 (40 samples)\n",
      "Train accuracy: 1.00\n",
      "Test accuracy: 0.85\n",
      "---------------\n",
      "Iteration #1 (60 samples)\n",
      "Train accuracy: 0.98\n",
      "Test accuracy: 0.86\n",
      "---------------\n",
      "Iteration #2 (80 samples)\n",
      "Train accuracy: 0.99\n",
      "Test accuracy: 0.86\n",
      "---------------\n",
      "Iteration #3 (100 samples)\n",
      "Train accuracy: 0.99\n",
      "Test accuracy: 0.87\n",
      "---------------\n",
      "Iteration #4 (120 samples)\n",
      "Train accuracy: 1.00\n",
      "Test accuracy: 0.88\n",
      "---------------\n",
      "Iteration #5 (140 samples)\n",
      "Train accuracy: 0.99\n",
      "Test accuracy: 0.88\n",
      "---------------\n",
      "Iteration #6 (160 samples)\n",
      "Train accuracy: 1.00\n",
      "Test accuracy: 0.88\n",
      "---------------\n",
      "Iteration #7 (180 samples)\n",
      "Train accuracy: 0.99\n",
      "Test accuracy: 0.89\n",
      "---------------\n",
      "Iteration #8 (200 samples)\n",
      "Train accuracy: 0.99\n",
      "Test accuracy: 0.89\n",
      "---------------\n",
      "Iteration #9 (220 samples)\n",
      "Train accuracy: 1.00\n",
      "Test accuracy: 0.89\n"
     ]
    }
   ],
   "source": [
    "import gc\n",
    "import torch\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "num_queries = 10\n",
    "\n",
    "\n",
    "def evaluate(active_learner, train, test):\n",
    "    y_pred = active_learner.classifier.predict(train)\n",
    "    y_pred_test = active_learner.classifier.predict(test)\n",
    "    \n",
    "    test_acc = accuracy_score(y_pred_test, test.y)\n",
    "\n",
    "    print('Train accuracy: {:.2f}'.format(accuracy_score(y_pred, train.y)))\n",
    "    print('Test accuracy: {:.2f}'.format(test_acc))\n",
    "    \n",
    "    return test_acc\n",
    "\n",
    "\n",
    "results_setfit = []\n",
    "results_setfit.append(evaluate(active_learner, train[labeled_indices], test))\n",
    "\n",
    "    \n",
    "for i in range(num_queries):\n",
    "    # ...where each iteration consists of labelling 20 samples\n",
    "    q_indices = active_learner.query(num_samples=20)\n",
    "\n",
    "    # Simulate user interaction here. Replace this for real-world usage.\n",
    "    y = train.y[q_indices]\n",
    "\n",
    "    # Return the labels for the current query to the active learner.\n",
    "    active_learner.update(y)\n",
    "    \n",
    "    # memory fix: https://github.com/UKPLab/sentence-transformers/issues/1793\n",
    "    gc.collect()\n",
    "    torch.cuda.empty_cache()\n",
    "\n",
    "    labeled_indices = np.concatenate([q_indices, labeled_indices])\n",
    "\n",
    "    print('---------------')\n",
    "    print('Iteration #{:d} ({} samples)'.format(i, len(labeled_indices)))\n",
    "    results_setfit.append(evaluate(active_learner, train[labeled_indices], test))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eeb9a63",
   "metadata": {},
   "source": [
    "## Plotting the Results\n",
    "\n",
    "With the previous results we can plot a [learning curve](https://en.wikipedia.org/wiki/Learning_curve_(machine_learning)) to visualize the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "278c1d13",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'matplotlib.pyplot' from '/home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/matplotlib/pyplot.py'>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABAwAAAK6CAYAAAC9oM4kAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAA9hAAAPYQGoP6dpAACdP0lEQVR4nOzdeVjVZf7/8ddh30EBwX1LEVEB99Jc0kJtNSvNyijrmzoz+TOnmZYZs5rJaXJmbG8mS21V0zTNvVxSXFKDVNwXFBFkUVZlOcvvD+QEoQJ25HDg+bguroH7fJb3caz4vM79vm+DxWKxCAAAAAAAoBwnexcAAAAAAADqHgIDAAAAAABQCYEBAAAAAACohMAAAAAAAABUQmAAAAAAAAAqITAAAAAAAACVEBgAAAAAAIBKCAwAAAAAAEAlLvYuoKELDQ1VQUGBWrVqZe9SAAAAAAANwKlTp+Tt7a20tLSrHscMAzsrKChQSUmJvcsAAAAAADQQJSUlKigoqPI4ZhjYWdnMgsTERDtXAgAAAABoCCIiIqp1HDMMAAAAAABAJQQGAAAAAACgEgIDAAAAAABQCYEBAAAAAACohMAAAAAAAABUQmAAAAAAAAAqITAAAAAAAACVEBgAAAAAAIBKXOxdAK5dSUmJTCaTvctAHeXs7CxXV1d7lwEAAADAQREYOKDc3FxlZmaqqKjI3qWgjnN3d1dQUJD8/PzsXQoAAAAAB0Ng4GByc3OVkpIiHx8fBQUFydXVVQaDwd5loY6xWCwqKSlRTk6OUlJSJInQAAAAAECNEBg4mMzMTPn4+KhFixYEBbgqT09P+fr66vTp08rMzCQwAAAAAFAjLHroQEpKSlRUVCR/f3/CAlSLwWCQv7+/ioqKVFJSYu9yAAAAADgQAgMHUrbAIQvZoSbK/r6wQCYAAACAmiAwcEDMLkBN8PcFAAAAwLUgMAAAAAAAAJUQGAAAAAAAgEoIDAAAAAAAQCUEBqiXjhw5ot///vfq3LmzvL295eHhoRYtWqhXr176/e9/r8WLF9dKHXPnzpXBYLjqV0BAgCRp48aNMhgMGjRoUK3UBgAAAABX42LvAgBb+/rrrzV27FgVFRUpMDBQ/fr1U3BwsM6fP6+EhAS9++67mj9/vkaNGvWb7pOUlKS2bduqdevWSkpKuuqx3t7euu+++y77mpeXV5X3Klu40GKx1LhOAAAAALgWBAaoV86ePatHH31URUVFmjp1qv72t7/Jw8OjwjG7d+/WokWLarWuoKAgzZ0796rH9O7dWwcOHKhWgAAAAAAA1xuBAeqVb7/9Vvn5+WrWrJlmzpx52WN69OihHj161HJlVfPy8lKnTp3sXQYAAAAASGINA9QzZ8+elSQFBwfX+Fyj0ajZs2dr0KBBaty4sdzd3dW2bVtNnDhRycnJFY6NjY1V27ZtJUknT56stC7BtbjcGgbTp0+vcL1f36eqVggAAAAAuFYOHRjs3LlTI0aMUEBAgLy9vdW3b18tXLiwRtc4c+aMJk+ebF0cLyQkRP3799enn34qk8lU6fjCwkK9+uqr6ty5szw8PNSoUSMNHz5ccXFxtnpb+A1atWolSdq3b5++//77ap+Xl5enW2+9VU8++aR2796tbt266a677pK7u7s++OADRUdHKz4+3np8//79rWsgeHt769FHH63wZStRUVEVrvfr+/j4+NjsXgAAAABQnsHioKuobdiwQTExMfLw8NCYMWPk6+urxYsX6+TJk5o5c6amTp1a5TWOHz+uPn36KCsrSzExMerWrZtyc3O1dOlSpaWlKTY2VnPmzLEeX1hYqCFDhmjr1q3q1q2bbrnlFmVnZ2vx4sW6cOGCFi9erLvvvrtG7yMiIkKSlJiYWOWxhYWFOnHihNq2bVupLx+l8vPz1alTJ6WkpMhgMGjgwIEaMmSIunfvrl69el1x5sFDDz2kL774QnfccYc++ugjNWnSxPrarFmzNGXKFHXo0EEHDhyQs7OzpOotejh37lw99thj1VoYcePGjRo8eLAGDhyojRs3Vnjttyx6yN8bAAAA4PqxWCzan5qrjYcytOlQhm4Jb6IJA9vbu6yrqu5zqEMGBkajUZ06ddLp06e1fft2RUVFSZJycnLUu3dvJSUl6fDhw2rduvVVrzNp0iS9//77mjVrliZPnmwdz87OVmRkpE6dOqWkpCTrdWbOnKlnn31W999/v7788kvrg+OxY8fUvXt3ubu769ixY/L19a32e7F1YHDn21uUkVdU7fvXFcG+7lr+h/42udahQ4f06KOPaseOHZVei4qK0lNPPaUnn3zS+v/fgQMHFBERoaZNm+rgwYOX/f/v9ttv18qVK7V8+XLdcccdkmoWGFzNhg0bNGjQIAIDAAAAwEHkXCjR5qMZpSHB4YwKz2C92jTSVxNusmN1Vavuc6hDLnq4fv16HTt2TI899pg1LJAkf39/vfDCC4qNjdW8efM0bdq0q17n+PHjkqQRI0ZUGA8ICFD//v31xRdfKDMz0xoYfPPNN5JK+8rLHjYlqX379nr88cc1a9YsLVq0qMoHxOspI69IabmFdrt/XRAWFqbt27frxx9/1IoVK7Rjxw799NNPysjIUEJCgiZOnKjFixdrxYoVcnNz08qVK2WxWDR8+PArhj2DBg3SypUrtXXrVmtgUBNX21YxNDS0xtcDAAAAUHvM5rJZBOnaeChD8cnZMpkv/2FezsUSlZjMcnV26BUAJDloYFD26ettt91W6bWYmBhJ0qZNm6q8TpcuXbRmzRqtXLmy0gyDuLg4hYaGqnPnztbxtLQ0SbIudlde2dj69evtGhgE+7rb7d6/xfWou3fv3urdu7ek0k/m4+Pj9cYbb2j+/Pn67rvv9Oabb+rZZ5+1BkcfffSRPvroo6teMyMj45pqqc62igAAAADqjuwLxfrhSKY2XZpFkJl/+Zncnq7Ouql9oAaFBWtgxyZqFVh/tkl3yMDgyJEjkqQOHTpUei00NFQ+Pj7WY67m2Wef1fLlyzVlyhStXr26whoGXl5eWrJkiTw9Pa3HBwUF6ejRozpx4kSFIEGSTpw4IUk6fPjwVe9ZVFSkoqJf/qKZzeZrXlX/cmw1rb++MRgM6t69u7788ktduHBBy5Yt09KlS/Xss8/KbDZLKm1XiIyMvOp1+vTpUxvlAgAAAKhlZrNF+87kaOOhDG08lK6E5GxdYRKB2gd7a1BYEw0KC1avNo3l4ep8+QMdnEMGBjk5OZJKWxAux8/Pz3rM1YSEhGjbtm16+OGHtWrVKq1evVqS5OnpqQkTJlR6eBw+fLi2b9+uV155RZ9//rm1LeHEiRPWxRGzs7Oves8ZM2bo5ZdfrjB2LVsA4trddtttWrZsmTIzMyVJLVu2lCT169dP77zzjj1LAwAAAFCLzhcU64cjpYsV/nAkQ5n5xZc9zsvNWTe1D7o0iyBYLRvXn1kEV+OQgYGtHD16VHfeead8fHy0efNmRUVFKTs7W5999pn+8pe/aM2aNdq8ebM1GJgyZYoWLFigBQsW6ODBgxV2SWjTpo327NkjJ6er96k8//zzeuaZZ6w/9+nTx6YzDBo6i8VS5Z/nqVOnJEktWrSQVBoEvfjii1q2bJlmzpxZ7YUB3dzcJJUuwnm9ubq6qqSkREajUS4uDfofWwAAAOCamc0W7UnJ0cZD6dp0OEM/X2UWQYcmPhoUFqxBYU3Us00jubvUz1kEV+OQTx5lMwuuNIsgNzdXjRo1qvI6sbGxOnnypI4fP25deM7Hx0fPPfeczp49q1mzZmn+/Pl66KGHJEm+vr6Ki4vTK6+8oiVLluidd95RkyZNNGHCBN1xxx0aMGBAhe34Lsfd3V3u7r/061cVMKBm3nvvPf3444966qmndNNNFVcmtVgs1v/fJGnMmDGSpOjoaI0aNUqLFy/Wvffeq/fee09t2rSpcG5BQYGWLl2qoUOHKiQkRFLpzBA3NzelpaXp3Llzaty48XV7Xy1atNCJEyeUmJhYZdsEAAAAgF+cKyjWD4dL2wx+OJKpcwWXn0Xg7easm274ZRZBi0YNYxbB1ThkYFC2dsGRI0fUo0ePCq+lpaUpPz/futjdleTl5SkuLk7du3e/7Cr1gwcP1qxZsxQfH28NDKTSHRT+/e9/69///neF48sWtOvZs+e1vCXYSElJiT755BN98sknCg4OVnR0tIKCgpSdna39+/dbtz98+OGHNX78eOt5c+bMUXZ2tlatWqWwsDBFRkaqbdu2slgsSkpK0s8//6zi4mIdOHDAGhi4urrqrrvu0qJFixQVFaX+/fvLy6v0XyqzZ8+26fsaNWqUZs6cqaFDh+qWW26x7ubw+uuvKzAw0Kb3AgAAAByZyWzRntPZpWsRHM7QntPZutLu5GEhvqUBQViwerZuLDcXPtAtzyEDg4EDB2rGjBlau3at9VPiMmvWrLEeczXFxaWpUlkf+6+VrYZffjbA1Xz++eeSVKke1K7x48erbdu2+v7777Vjxw7t379fZ8+elYuLi5o1a6YHH3xQ48aN07Bhwyqc5+vrq7Vr12rBggX67LPPtHv3biUkJMjPz09NmzbVQw89pLvuukvt27evcN5///tfBQYGatWqVVq0aJFKSkok2T4wePXVV+Xk5KSvv/5aS5cutf79/ctf/kJgAAAAgAYvM7/o0iyCDG0+kqHzF0oue5yPu4v63RCoQWFNNLBjsJoFeF72OJQyWCxXylrqLqPRqLCwMKWkpGj79u2KioqSVNqi0Lt3byUlJenQoUPWaeWpqanKyclR06ZNKyyU2KlTJx06dEgffvihnnjiCet4dna2brzxRh08eFDr1q3T0KFDra/l5ubKz8+vQj3/+c9/9Mwzz2jkyJH6+uuva/ReIiIiJEmJiYlVHltYWKgTJ06obdu21e6zB/h7AwAAgPrGZLYoITlbmw6la+PhDO1NybniLIJOob4aGBasQR2bqEfrRswiUPWfQx1yhoGLi4tmz56tmJgYDRgwQGPGjJGvr68WL16skydPaubMmRV60J9//nnNmzdPc+bMUWxsrHX8P//5j+666y49+eSTmj9/vqKjo3X+/HktW7ZMGRkZGjVqVIWwQJKaN2+uwYMHq0OHDjIYDNq4caN2796tnj176qOPPqqlPwEAAAAAaFgy8i7NIjhcOosg+wqzCHzdXdS/Q9laBE0U6s+HZtfKIQMDqXSNgS1btuill17SggULVFJSoq5du+r111/X6NGjq3WN4cOHa+vWrXrjjTe0ZcsWbdq0SR4eHgoPD9e0adM0ceLESuc8/PDD2rBhg77//nsZDAZ17NhRb7zxhv7whz9Uu30BAAAAAHB1RpNZCcllaxGka19K7hWPDW/qV7qjQcdgdW/dSK7OzCKwBYdsSahPaEnA9cbfGwAAADiK9LxCbbq0WOGWI5nKuXiFWQQeLrq5Q5AGdWyigWHBCvHj99yaqNctCQAAAAAAx2c0mRWfnK2Nh9K18VCGEs9ceRZBRLNLswjCmii6ZYBcmEVw3REYAAAAAABqzdncslkE6dp8JFN5hcbLHufn4aKbO5a2GQzsGKwmzCKodQQGAAAAAIDrpsRk1k8nz2vjpW0PD6ReeRZBl+Z+GtSxiQaFBSuKWQR2R2AAAAAAALCptJxCbTpc2maw5Uim8oouP4vA39NVAy7NIri5Y5Ca+DKLoC4hMHBArFOJmuDvCwAAAK63EpNZu5LOa+PhdG06lKGDaXlXPLZbC//SNoOwJopqGSBnJ0MtVoqaIDBwIE5OpdNxTCaTnSuBIyn7+1L29wcAAACwhdSci6VbHh5KV9zRLOVfYRZBgJerBnQI1qCwYA3oGKwgH7ajdxQEBg7E1dVVzs7Ounjxonx8fOxdDhzExYsX5ezsLFdXV3uXAgAAAAdWbDRrV9I5bbq0FsGhs5efRWAwSN1aBGhQx9KQoFsLZhE4KgIDB2IwGOTl5aWcnBw1btxYzs7O9i4JdZzJZFJOTo68vLxkMPAvaQAAANRMSvZF65aHW49mqqD48rOdG3u7aUCHIA0Ka6KbOwQpkFkE9QKBgYNp0qSJkpKSdPLkSTVu3Fju7u48CKISi8WioqIinTt3TmazWU2aNLF3SQAAAHAARUZT6VoEl0KCI+n5lz3OYJAiWwRoUFiwBoU1Udfm/swiqIcIDByMm5ubWrRooczMTKWmptq7HNRx3t7eCg0NlZubm71LAQAAQB2VfO6CNh7O0KZD6dp6LEsXrjCLINDbTQM7BmtgWLAGdAhWI29+x6zvCAwckJeXl1q1aiWj0Sij8fILiwAuLi5yceEfcQAAgIaoxGRWXqFRuRdLlFtYotyLxkv/W1I6XliirIJi7TiepWMZBZe9hpNBimoZoEFhTTQoLFhdmvnLiVkEDQpPEw6MB0IAAACgfioymi77kF9+LLewpFwoUDEcuFhybTurBfm4aUDH0jaDAR2CFODFLIKGjKdNAAAAALAhi8WiwhKz8gpLH+BzKjzkG0vHfzVWGgr88n2R0VwrtToZpO6tGlnXIujc1I9ZBLAiMAAAAACAciwWiy4Umyp8on/lh/zKY7mFJSoxWWq9bm83Z/l6uMrP00V+Hq7y83SVn4eL/Dxd5etRfqz0GF8PV7UN9Ja/F9tv4/IIDAAAAADUK2azRfnFxl+m6//qYf6yvf2Xvi/7lN9krv0Hfl/3Xz/cV37w9/O49Hq5B38/D1f5eLjI1dmp1mtG/UZgAAAAAKDOO3I2TwnJ2ZV69S/36X9ekVGWWn7eNxh+eeD/9cN+xU/3L3PMpQd+tiVEXUNgAAAAAKBOOptbqOU/n9GS+BQlnsm9rvdydjJUnr5/mU/5fX/94H/pe283F3r/Ue8QGAAAAACoM/KLjFq9L01L41O09VimqtsZ4OpsqKJv36VS/375QMDLzVkGAw/8QHkEBgAAAADsqsRk1uYjGVoSf0br9qepsKTyDgGRLfw1vGtTNfX3sD74+1962Pf1cJWHqxMP/ICNERgAAAAAqHUWi0UJydlaGp+i5XtSda6guNIxLRt7amRUc90d3Vztg33sUCXQsBEYAAAAAKg1SZkFWpqQoqXxKUrKulDp9QAvV93RralGRjdX91aNmDUA2BGBAQAAAIDrKiu/SN/uSdXShBTFn8qu9Lqbi5Nu7RyikVHNNaBjsNxc2B4QqAsIDAAAAADY3MVik9YdOKul8Sn64XCGjL9avdBgkG5sF6h7optrWJdQ+Xm42qlSAFdCYAAAAADAJkxmi7Ydy9KS+BSt3peqgmJTpWM6hfpqZHRz3RXVTE39Pe1QJYDqIjAAAAAAcM0sFov2p+ZqaXyKvkk4o/S8okrHhPp56O7oZronqrnCm/rZoUoA14LAAAAAAECNnT5/Qd8knNHS+BQdSc+v9Lqvu4tGdG2qu6ObqW/bQDk5sXgh4GgIDAAAAABUS86FEq3cl6ol8Sn68cS5Sq+7Ohs0KKyJRkY31y2dmsjD1dkOVQKwFQIDAAAAAFdUZDRpw8EMLY1P0fqD6So2mSsd07N1I90T3Vy3d22qRt5udqgSwPVAYAAAAACgArPZop1J57Q04YxW7Dmj3EJjpWPaBXvr3ujmujuquVo29rJDlQCuNwIDAAAAAJKkI2fztOTS4oUp2RcrvR7k4667IptpZHRzdWnuJ4OBdQmA+ozAAAAAAGjA0nMLteznM1oSn6LEM7mVXvd0ddawLqG6J7q5+rUPlIuzkx2qBGAPBAYAAABAA5NfZNSafWlampCiuKOZMlsqvu5kkG7uEKyR0c11a+cQebvz2AA0RPyTDwAAADQAJSazNh/J0NL4M1q7P02FJZUXL4xs4a97opvrjm7NFOzrbocqAdQlBAYAAABAPWWxWJSQnK2l8SlavidV5wqKKx3TsrGnRkY1193RzdU+2McOVQKoqwgMAAAAgHomKbNASxNStDQ+RUlZFyq9HuDlqju6NdXI6Obq3qoRixcCuCwCAwAAAKAeyMov0oq9qVoSn6L4U9mVXndzcdKt4SG6J7q5BnYMlpsLixcCuDoCAwAAAMBBXSw26bsDZ7U0PkWbDmfI+KvVCw0G6cZ2gbonqrmGdQ2Vn4ernSoF4IgIDAAAAAAHYjJbtO1YlpbEp2j1vlQVFJsqHdMp1Fcjo5vrrqhmaurvaYcqAdQHBAYAAABAHWexWLQ/NVdL41P0TcIZpecVVTom1M9Dd0c30z1RzRXe1M8OVQKobwgMAAAAgDoqJfuivrm0eOHhs/mVXvd1d9GIrk11d3Qz9W0bKCcnFi8EYDsEBgAAAEAdknOxRKsuLV6448S5Sq+7Ohs0KKyJRkY31y2dmsjD1dkOVQJoCAgMAAAAADsrMpq04WCGlsanaP3BdBWbzJWO6dm6ke6Jbq7buzZVI283O1QJoKEhMAAAAADswGy2aNfJ81oSn6IVe84ot9BY6Zh2wd4aGdVcd0c1V6tALztUCaAhIzAAAAAAatHR9DwtiU/R0vgzSsm+WOn1IB933RnZVCOjm6trc38ZDKxLAMA+CAwAAACA6yw9t1DLfj6jpQkp2peSW+l1T1dnDesSqnuim6tf+0C5ODvZoUoAqIjAAAAAALgO8ouMWrMvTUsTUhR3NFNmS8XXnQzSzR2CNTK6uW7tHCJvd341B1C38G8lAAAAwEYsFot2nDinhbuStWpvmi6WmCod062Fv+6Jaq47Ipuqia+HHaoEgOohMAAAAAB+o7ScQi3+6bQW7krWyawLlV5v0chTI6NLFy+8oYmPHSoEgJojMAAAAACuQbHRrO8OnNXCXcn64XBGpZYDPw8X3RnZTPd2b67urRqxeCEAh0NgAAAAANTAwbRcLdx5WksTUnSuoLjS6/1vCNL9PVsoJiJUHq7OdqgQAGyDwAAAAACoQs7FEi3/+YwW7krWntM5lV5vHuCp+3q00H09WqhlYy87VAgAtkdgAAAAAFyG2WzR9hNZWrgzWav2panIaK7wupuLk2IiQjW6Z0vd1D5QTk60HACoXwgMAAAAgHJSsi9q8e7T+mp3spLPXaz0epfmfnqgZ0vdFdlMAV5udqgQAGoHgQEAAAAavCKjSev2n9XCXae1+UiGLL9awNDf01Ujo5vr/p4tFNHM3z5FAkAtIzAAAABAg7X/TK4W7krW0oQUZV8oqfCawVC6gOHoXi01NDyEBQwBNDgEBgAAAGhQci6U6JufU7RwV7L2peRWer1FI0890LOlRvVooeYBnnaoEADqBgIDAAAA1Htms0Vbj2Vp4a5krU5MU/GvFjB0d3HS8C6heqBnS/VtxwKGACARGAAAAKAeO33+ghbtPq2vdp1WSnblBQy7tfDXAz1b6s7IZvL3dLVDhQBQdxEYAAAAoF4pLDFpTWKavtp1WnHHMistYNjIy1Ujo1vo/p4tFN7Uzz5FAoADIDAAAABAvbAvJad0AcP4FOUWGiu85mSQBnQM1gM9W2pIeBO5u7CAIQBUhcAAAAAADut8QbG+SUjRwl2ntT+18gKGrQO99EDPlrq3e3M19WcBQwCoCQIDAAAAOBST2aK4o5lasCtZ6xLPqthUcQFDD1cnjejaVA/0bKnebRqzgCEAXCMCAwAAADiEU1kXtGh3shbtPq0zOYWVXo9qGaAHerbUHZFN5efBAoYA8FsRGAAAAKDOKiwxafW+NC3claytx7IqvR7o7aaR0c11f8+WCgv1tUOFAFB/Odm7gN9i586dGjFihAICAuTt7a2+fftq4cKFNbrGmTNnNHnyZHXu3Fne3t4KCQlR//799emnn8pkMlU63mg06uOPP9aNN96o4OBg+fr6qnPnzvrTn/6ktLQ0W701AACABstisWjP6Wy9uGSvev39O/2/BQkVwgIng3RLpyb64OEe2vb8EP3ljs6EBQBwHRgsll9vNOMYNmzYoJiYGHl4eGjMmDHy9fXV4sWLdfLkSc2cOVNTp06t8hrHjx9Xnz59lJWVpZiYGHXr1k25ublaunSp0tLSFBsbqzlz5lQ4Z9SoUfr66691ww03aNiwYXJ3d9f27dsVFxenpk2b6qefflJoaGi130dERIQkKTExsWZ/AAAAAPXMuYJiLYlP0Ve7knUwLa/S622DvHV/zxYa1b2FQvw87FAhANQP1X0OdcjAwGg0qlOnTjp9+rS2b9+uqKgoSVJOTo569+6tpKQkHT58WK1bt77qdSZNmqT3339fs2bN0uTJk63j2dnZioyM1KlTp5SUlGS9zo8//qg+ffqod+/e2rJli1xdf+mNmzx5st566y29/PLLmjZtWrXfC4EBAABoyExmi344kqGvdiVr3f6zKjFV/NXU09VZt3crXcCwV5tGMhhYwBAAfqvqPoc6ZEvC+vXrdezYMY0dO9YaFkiSv7+/XnjhBRUXF2vevHlVXuf48eOSpBEjRlQYDwgIUP/+/SVJmZmZlY4fOnRohbBAku644w5JUkZGRs3fEAAAQAOTlFmgN9YcVL9/rNdjc3Zq5d60CmFB91YBen1UV+38y1DNvD9Svds2JiwAgFrmkIsebty4UZJ02223VXotJiZGkrRp06Yqr9OlSxetWbNGK1eurDTDIC4uTqGhoercubN1vCyF+e677zR9+vQKocG3334rSRoyZEjN3xAAAEADcKHYqFV7Sxcw3HHiXKXXg3zcNKp7C93fs4VuaMKaBABgbw4ZGBw5ckSS1KFDh0qvhYaGysfHx3rM1Tz77LNavny5pkyZotWrV1dYw8DLy0tLliyRp6en9fiuXbtq8uTJevPNN9W5c2cNHz5c7u7u2rZtm3bv3q2XX35Z99xzz1XvWVRUpKKiIuvPZrOZtBwAANRbFotFCcnZWrjrtJb/fEb5RcYKrzs7GTQ4rIke6NlCgzs1kauzQ06ABYB6ySEDg5ycHEmlLQiX4+fnZz3makJCQrRt2zY9/PDDWrVqlVavXi1J8vT01IQJExQZGVnpnFmzZqlt27Z69tln9fbbb1vH77zzTt17771V3nPGjBl6+eWXK4wFBwdXeR4AAIAjycwv0pKfUrRwV7KOpOdXer1dsLdG92ypkd2bq4kvCxgCQF3kkIGBrRw9elR33nmnfHx8tHnzZkVFRSk7O1ufffaZ/vKXv2jNmjXavHmznJ2dJZXOBpgwYYK+/PJLvf3227r77rvl5eWluLg4Pf300+rbt682bNigXr16XfGezz//vJ555hnrz3369GGGAQAAqBeMJrM2Hc7Qwl3J+v5AuozmigsYers5645uzfRArxbq3ooFDAGgrnPIwKBsZsGVZhHk5uaqUaNGVV4nNjZWJ0+e1PHjx61bIfr4+Oi5557T2bNnNWvWLM2fP18PPfSQJOnjjz/Whx9+qDfffFNPPfWU9TrDhw/XokWLFBUVpRdeeEHr1q274j3d3d3l7u5u/dnJiWl3AADAsR3PyNdXu09r8e7TSs8rqvR6rzaNdH/Plrq9a1N5uzvkr58A0CA55L+xy9YuOHLkiHr06FHhtbS0NOXn56t3795XvUZeXp7i4uLUvXt3a1hQ3uDBgzVr1izFx8dbA4NVq1ZZX/u1yMhINWrUSPHx8df0ngAAABxJQZFRK/am6qtdydqZdL7S60183TWqRwvd36OF2gX72KFCAMBv5ZCBwcCBAzVjxgytXbtWY8aMqfDamjVrrMdcTXFxsaSK2yaWV7Y9YvnZAGXnXG7rxKKiIuXl5alJkybVfBcAAACOxWKx6KdT57Vw52l9u+eMCopNFV53cTJoSHgTPdCzpQZ2DJYLCxgCgENzyH+LDxkyRO3atdMXX3yhhIQE63hOTo5ee+01ubm5ady4cdbx1NRUHTx4sEILQ2BgoMLCwnTq1CnNnj27wvWzs7M1c+ZMSRVnE/Tr10+S9Nprr1XY6UCSpk+fLqPReNnZBwAAAI4sPa9Q/910TEP/vUmj3t+mBbuSK4QFNzTx0YsjwrXt+SH67yM9NSQ8hLAAAOoBg8VisVR9WN2zYcMGxcTEyMPDQ2PGjJGvr68WL16skydPaubMmZo6dar12NjYWM2bN09z5sxRbGysdXzVqlW66667ZDQaNWTIEEVHR+v8+fNatmyZMjIyNGrUKC1atMh6fG5urvr06aODBw+qTZs2GjZsmDw9PRUXF6cff/xRwcHB2r59u9q1a1ft9xERESFJSkxM/O1/KAAAADZSYjJrw8F0Ldx1WhsOpcv0qwUMfdxddGdkUz3Qs6WiWgawgCEAOJDqPoc6ZEuCVPrJ/5YtW/TSSy9pwYIFKikpUdeuXfX6669r9OjR1brG8OHDtXXrVr3xxhvasmWLNm3aJA8PD4WHh2vatGmaOHFiheP9/Py0fft2vf766/rmm280d+5cmUwmtWjRQhMmTNCLL76oFi1aXI+3CwAAUCuOpufrq13JWvxTijLzKy9g2KdtYz3Qs6WGdw2Vl5vD/ioJAKgGh51hUF8wwwAAgPrJYrHIZLbIaLbIfOl7s1kyXfreZLbIZLHI/KvvjZd+tp5jschkloxms/X88ueYfnV82fdGc/njVPp9+XuXP8dikclkUXxytnafrLyAYYifu+7r0UL392ipNkHedvjTBADYUr2fYQAAABquzPwiLY1P0bmC4nIP0LI+AFsfli2WKh6UJZPZXOlh3myxyGiq+EBtLvdgbbJUvJ/1Qb/ccY7+kYyrs0FDw0P0QK+WGtAhWM5OtBwAQENDYAAAABzKhoPp+uNXPyuroNjepdRLYSG+eqBXS90T1UyBPu5VnwAAqLcIDAAAgEMoLDHpH6sOau7WpOt+LyeD5OxkKP0yGOT0q+9dnAxyMhisx/xyvJOcnfTLOeWOdy53jpOh3JiTQc4GWY93ca54nLWO8rUYDKX3uXS/irVcqs96XMXvS7/KnVNuPMDLTe2DvVnAEAAgicAAAAA4gMNn8/T0l/E6mJZnHbulUxM93q+tXJx/9XBtMMjJSXL51cN02f+Wf5guHwSUf/jngRkAAAIDAABQh1ksFn22/aT+tuKAioxmSZKbi5P+cnu4Hunbmgd7AACuIwIDAABQJ50rKNafFu3RdwfOWsfCQnz11oPRCgv1tWNlAAA0DAQGAACgztlyJFPPLExQel6RdezRG1vr+RHh8nB1tmNlAAA0HAQGAACgzig2mvWvtYf03x+OW8cae7vpjfu6aUh4iB0rAwCg4SEwAAAAdcLxjHw9PT9e+1JyrWM3dwjSv+6PVBM/DztWBgBAw0RgAAAA7MpisWjhrmRNX7ZfF0tMkiRXZ4P+PKyTHu/XVk5OLGwIAIA9EBgAAAC7yblQoueX7NHKvWnWsXbB3nprTLS6NPe3Y2UAAIDAAAAA2MWO41masiBBZ3IKrWMP9m6lv94RLi83fkUBAMDe+K8xAACoVSUms9787oje3XhUFkvpmL+nq14f1VXDujS1b3EAAMCKwAAAANSaU1kX9PT8eCUkZ1vH+rZrrP+MjlJTf0/7FQYAACohMAAAALViSfxp/XVpovKLjJIkFyeDnrmto54a0F7OLGwIAECdQ2AAAACuq9zCEk1buk9LE85Yx1oHeunNMdGKahlgv8IAAMBVERgAAIDrZvfJ85o8P16nz1+0jo3q3kIv3x0hH3d+DQEAoC7jv9QAAMDmTGaL3t1wVG9+f0Qmc+nKhr7uLvr7vV11V2QzO1cHAACqg8AAAADYVEr2RU2Zn6Afk85Zx3q0bqRZo6PUsrGXHSsDAAA1QWAAAABs5ts9Z/T813uVV1i6sKGTQXp6SAf9fvANcnF2snN1AACgJggMAADAb1ZQZNT0ZYn6avdp61jzAE+9OSZKPds0tmNlAADgWhEYAACA32TP6WxNnp+gE5kF1rE7I5vpb/d0kb+nqx0rAwAAvwWBAQAAuCZms0X/23xcM9cckvHSwobebs565e4uurd7cxkMBjtXCAAAfgsCAwAAUGNpOYV6ZmGCth7Lso5FtvDXm2Oi1SbI246VAQAAWyEwAAAANbI2MU1/XrxH5y+USJIMBmniwPaacmtHubKwIQAA9QaBAQAAqJaLxSb9bcV+fb7jlHUs1M9D/xkdpRvbB9qxMgAAcD0QGAAAgCrtP5Orp+fH62h6vnVsWESoZtzbVY283exYGQAAuF4IDAAAwBVZLBbNiUvSP1YdVLHJLEnycHXSS3dGaEyvlixsCABAPUZgAAAALisjr0h//OpnbTqcYR3r3NRPbz0YrRua+NixMgAAUBsIDAAAQCUbDqXr2a9+VmZ+sXXsyZvb6o8xYXJ3cbZjZQAAoLYQGAAAAKvCEpNeX31Qc+KSrGNBPu769wORGtAx2H6FAQCAWkdgAAAAJElHzubpD1/G62BannXslk5N9M/7uinIx92OlQEAAHsgMAAAoIGzWCz6bMcp/e3b/Soyli5s6ObipBdHhGvcja1Z2BAAgAaKwAAAgAbsXEGx/rRoj747cNY61jHER289GK1OoX52rAwAANgbgQEAAA1U3NFMTVmQoPS8IuvYoze21vMjwuXhysKGAAA0dAQGAAA0MMVGs/619pD+t/m4LJbSscbebnrjvm4aEh5i3+IAAECdQWAAAEADcjwjX5PnJ2hvSo517OYOQfrX/ZFq4udhx8oAAEBdQ2AAAEADYLFY9NWu03ppWaIulpgkSa7OBv0pppPG928rJycWNgQAABURGAAAUM/lXCjRC0v2asXeVOtYu2BvvTUmWl2a+9uxMgAAUJcRGAAAUI/tOJ6lKQsSdCan0Dr2YO+W+usdneXlxq8BAADgyvhNAQCAeqjEZNZb3x/RuxuOynxpYUN/T1f9496uGt61qX2LAwAADoHAAACAeuZU1gVNXhCv+FPZ1rG+7Rrr3w9EqVmAp/0KAwAADoXAAACAemRpfIr+snSf8ouMkiRnJ4OeubWjJgxsL2cWNgQAADVAYAAAQD2QV1iiad8kakl8inWsVWMvvTkmStGtGtmxMgAA4KgIDAAAcHA/nTqvyfPjlXzuonXs3u7N9crdXeTjzn/qAQDAteG3CAAAHJTJbNF7G45q1vdHZLq0sqGvu4v+NrKL7o5qbufqAACAoyMwAADAAaVkX9SU+Qn6MemcdaxH60aaNTpKLRt72bEyAABQXxAYAADgYFbsSdXzX+9RbmHpwoZOBukPt3TQH265QS7OTnauDgAA1BcEBgAAOIiCIqNeXp6ohbtOW8eaB3hq1pgo9WrT2I6VAQCA+ojAAAAAB7D3dI6enh+vE5kF1rE7ujXV30d2lb+nqx0rAwAA9RWBAQAAdZjZbNGHm49r5tpDKjGVLmzo5easV+7uolHdm8tgMNi5QgAAUF8RGAAAUEedzS3UMwsTFHc0yzrWrYW/3hwTrbZB3nasDAAANAQEBgAA1EHr9p/Vnxb9rPMXSiRJBoM0YWB7TRnaUW4uLGwIAACuPwIDAADqkIvFJv195X59tv2UdSzUz0P/Hh2pm9oH2bEyAADQ0BAYAABQRxxIzdXTX8brSHq+dSwmIkT/uLebGnm72bEyAADQEBEYAABgZxaLRXO3JmnGqoMqNpolSR6uTpp2R4Qe7N2ShQ0BAIBdEBgAAGBHGXlFenbRz9p4KMM61rmpn956MEo3NPG1Y2UAAKChIzAAAMBONh5K1x+/+lmZ+cXWsSf6t9Wzw8Lk7uJsx8oAAAAIDAAAqHWFJSb9c/UhfRx3wjoW5OOufz0QqYEdg+1YGQAAwC8IDAAAqEVHzubp6fkJOpCaax0bHBasN+6PVJCPux0rAwAAqIjAAACAWmCxWPT5jlN69dv9Krq0sKGbi5NeGN5Jj97UhoUNAQBAnUNgAADAdZZfZNTUhQlak3jWOtYxxEdvPRitTqF+dqwMAADgyggMAAC4js5kX9Tjc3fqYFqedWzcja31wohwebiysCEAAKi7CAwAALhO9pzO1vh5u5SRVyRJ8vd01b/uj9TQziF2rgwAAKBqBAYAAFwHaxLTNHl+vApLStcraB3opY9je6l9sI+dKwMAAKgeJ3sX8Fvs3LlTI0aMUEBAgLy9vdW3b18tXLiwRtc4c+aMJk+erM6dO8vb21shISHq37+/Pv30U5lMpgrHTp8+XQaD4apf48ePt+VbBAA4GIvFov/9cEwTPtttDQt6tWmkJZP6ERYAAACH4rAzDDZs2KCYmBh5eHhozJgx8vX11eLFizV69GglJydr6tSpVV7j+PHj6tOnj7KyshQTE6M777xTubm5Wrp0qcaNG6f169drzpw51uMHDRp0xWvNnj1bKSkpiomJscXbAwA4oBKTWS8tS9QXO05Zx+6JaqbX7+smdxfWKwAAAI7FYLFYLPYuoqaMRqM6deqk06dPa/v27YqKipIk5eTkqHfv3kpKStLhw4fVunXrq15n0qRJev/99zVr1ixNnjzZOp6dna3IyEidOnVKSUlJVV7n7NmzatGihfz9/XXmzBm5ublV+71ERERIkhITE6t9DgCg7sktLNHvPv9Jm49kWsf+39AOmjykA1smAgCAOqW6z6EO2ZKwfv16HTt2TGPHjrWGBZLk7++vF154QcXFxZo3b16V1zl+/LgkacSIERXGAwIC1L9/f0lSZmZmpfN+bd68eTIajXrkkUdqFBYAAOqH5HMXNOq9rdawwM3ZSbNGR+n/De1IWAAAAByWQwYGGzdulCTddtttlV4rawnYtGlTldfp0qWLJGnlypUVxrOzsxUXF6fQ0FB17ty5yut89NFHkqQnnniiymMBAPVL/KnzGvlenI6k50uSGnm56vMn++ie6OZ2rgwAAOC3ccg1DI4cOSJJ6tChQ6XXQkND5ePjYz3map599lktX75cU6ZM0erVq9WtWzfrGgZeXl5asmSJPD09r3qNzZs36/Dhw+rbt691WsfVFBUVqaioyPqz2Wzm0ycAcFAr9qTqmYUJKjKWLm7YLshbH8f2UpsgbztXBgAA8Ns5ZGCQk5MjqbQF4XL8/Pysx1xNSEiItm3bpocfflirVq3S6tWrJUmenp6aMGGCIiMjq7xGTWcXzJgxQy+//HKFseDg4GqdCwCoGywWi97beExvrDlkHevbrrE+eLiHArxoTQMAAPWDQ7Yk2MrRo0fVr18/ZWRkaPPmzcrLy1NycrKmTZumV199VUOGDKm0tWJ5ubm5+uqrr+Tj46PRo0dX657PP/+8cnJyrF+dOnVSUFCQrd4SAOA6Kzaa9efFeyqEBff1aKFPHu9DWAAAAOoVh5xhUDaz4EqzCHJzc9WoUaMqrxMbG6uTJ0/q+PHjCg0NlST5+Pjoueee09mzZzVr1izNnz9fDz300GXPnz9/vi5cuKDx48fLx6d6e2u7u7vL3d3d+rOTU4PObADAoeRcKNGEz3Zr2/Es69izMWGaNKg97WUAAKDeccin1bK1Cy63TkFaWpry8/Mvu75BeXl5eYqLi1N4eLg1LChv8ODBkqT4+PgrXmP27NmSWOwQABqCk1kFGvl+nDUscHNx0jtjo/W7wTcQFgAAgHrJIQODgQMHSpLWrl1b6bU1a9ZUOOZKiouLJV1528SMjAxJqjAboLy9e/dq586dioiIUN++fatXOADAIe1KOqeR723V8YwCSVKgt5vm/19f3dGtmZ0rAwAAuH4cMjAYMmSI2rVrpy+++EIJCQnW8ZycHL322mtyc3PTuHHjrOOpqak6ePBghRaGwMBAhYWF6dSpU9aZAmWys7M1c+ZMSb/MNPi1ssUOx48fb6u3BQCog75JSNHYD3foXEFp0HxDEx8t/V0/dW9VdesbAACAIzNYLBaLvYu4Fhs2bFBMTIw8PDw0ZswY+fr6avHixTp58qRmzpypqVOnWo+NjY3VvHnzNGfOHMXGxlrHV61apbvuuktGo1FDhgxRdHS0zp8/r2XLlikjI0OjRo3SokWLKt27uLhYzZo1U15ens6cOaPAwMBrfh9lWzEmJiZe8zUAALZnsVj01vdH9Z/vDlvH+t8QpHcf6i5/T1c7VgYAAPDbVPc51CEXPZRKP/nfsmWLXnrpJS1YsEAlJSXq2rWrXn/99WrvWDB8+HBt3bpVb7zxhrZs2aJNmzbJw8ND4eHhmjZtmiZOnHjZ85YuXaqsrCw98MADvyksAADUTUVGk55fvFdfx6dYxx7s3VKv3N1Frs4OOTkPAACgxhx2hkF9wQwDAKhbzhcU66lPd+vHpHOSJINBem5YJ/3fgHYsbggAAOqFej/DAAAAWzueka/H5+5UUtYFSZKHq5NmjY7SsC5N7VwZAABA7SMwAABA0vbjWZrw2W5lXyiRJAX7umv2uJ6KbBlg38IAAADshMAAANDgLd59Ws99vUclptIuvU6hvvootpeaB3jauTIAAAD7ITAAADRYFotF/153WG+vP2odG9gxWO+MjZavBzshAACAho3AAADQIBWWmPTsoj1a/vMZ69gjfVvrpTs7y4WdEAAAAAgMAAANT1Z+kZ78ZJd+OpUtqXQnhL/c3lmP92vDTggAAACXEBgAABqUo+l5emzuTiWfuyhJ8nJz1ltjojW0c4idKwMAAKhbCAwAAA1G3NFMTfhst/IKjZKkED93ffRoL3Vp7m/nygAAAOoeAgMAQIOwYOcpvbhkn4zm0p0QOjf100exPdXUn50QAAAALofAAABQr5nNFv1zzSF9sOmYdWxoeBO9OSZa3u78ZxAAAOBK+E0JAFBvXSw26ZmFCVq1L8069li/NvrL7Z3l7MTihgAAAFdDYAAAqJfS8wr15Ce79XNytiTJySBNvytC425sY9e6AAAAHAWBAQCg3jmUlqfH5+5USnbpTgjebs5656HuGhzWxM6VAQAAOA4CAwBAvbLpcIZ+//lPyisq3Qmhmb+HPortpfCmfnauDAAAwLEQGAAA6o3Ptp/US8sSZbq0E0LX5v766NGeauLnYefKAAAAHA+BAQDA4ZnMFs1YeUCzt5ywjsVEhOg/o6Pk5cZ/6gAAAK4Fv0UBABzahWKjJs9P0Lr9Z61j/zegnZ4b1klO7IQAAABwzQgMAAAO62xuocbP26l9KbmSJGcng169u4vG9mll58oAAAAcH4EBAMAh7T+Tq/Hzdio1p1CS5Ovuovce7q6bOwTbuTIAAID6gcAAAOBw1h88qz98Ea+CYpMkqXmAp+Y81ksdQ3ztXBkAAED9QWAAAHAoc+NO6JVv9+vSRgiKahmgD8f1VLCvu30LAwAAqGcIDAAADsFktujVb/dr7tYk69jtXZvqXw9EysPV2X6FAQAA1FMEBgCAOi+/yKinv4zX+oPp1rHfDW6vqbeGsRMCAADAdUJgAACo085kX9T4ebt0ILV0JwQXJ4Neu7erHujZ0s6VAQAA1G8EBgCAOmvv6RyNn7dT6XlFkiQ/Dxd98EgP3dQ+yM6VAQAA1H8EBgCAOmltYpomz0/QxZLSnRBaNfbSnMd6qX2wj50rAwAAaBgIDAAAdYrFYtFHW07o7ysPyHJpJ4SerRvpf+N6qrG3m32LAwAAaEAIDAAAdYbRZNa0ZYn6Yscp69jdUc30+qhu7IQAAABQywgMAAB1Qm5hiX73+U/afCTTOjZ5SAf9v6EdZDCwEwIAAEBtIzAAANhd8rkLGj9vpw6fzZckuTk76fX7umpkdAs7VwYAANBwERgAAOwqITlbT8zbqcz8YklSIy9X/feRnurdtrGdKwMAAGjYCAwAAHazcm+qpixIUJHRLElqG+StObG91CbI286VAQAAgMAAAFDrLBaLPth0XK+vPmgd69O2sf77SA8FeLETAgAAQF1AYAAAqFUlJrP+smSfFuxKto6N6t5CM+7tKjcXJztWBgAAgPIIDAAAtSbnQokmfr5bW49lWcf+eFtH/W7wDeyEAAAAUMcQGAAAasWprAt6bO6POpZRIElyc3HSzPsjdVdkMztXBgAAgMshMAAAXHe7T57Tk5/s1rmC0p0QAr3d9L9xPdWjdSM7VwYAAIArITAAAFxXy34+oz9+9bOKL+2E0D7YW3Nie6tVoJedKwMAAMDVEBgAAK4Li8Wid9Yf1b/WHbaO9bshUO891EP+nq52rAwAAADVQWAAALC5IqNJz3+9V1//lGIdG92zpf42sotcndkJAQAAwBEQGAAAbOp8QbGe+my3fjxxzjr23PBOempAO3ZCAAAAcCAEBgAAmzmRWaDH5+7UiczSnRDcXZw0a3SUhndtaufKAAAAUFMEBgAAm9hxPEtPfbZb2RdKJElBPu6a/WhPRbUMsG9hAAAAuCYEBgCA3+zrn07rz4v3qMRkkSR1DPHRx7G91KIROyEAAAA4KgIDAMA1s1gs+s+6w3pr/VHr2ICOwXpnbLT8PNgJAQAAwJERGAAArklhiUl/WrRHy34+Yx17qE8rvXxXhFzYCQEAAMDhERgAAGosK79IT326W7tOnpckGQzSiyPCNb5/W3ZCAAAAqCcIDAAANXI0PV+Pz92pU+cuSJI8XZ315pgo3RYRaufKAAAAYEsEBgCAatt6NFMTPtut3EKjJKmJr7s+erSXurbwt3NlAAAAsDUCAwBAtSzclawXvt4ro7l0J4Twpn76OLanmvp72rkyAAAAXA82WZVq0aJFKikpscWlAAB1jNls0eurD+pPi/ZYw4JbOjXRVxNuJCwAAACox2wSGDzwwANq1qyZpkyZoj179tjikgCAOqCwxKQ/fBmv9zces47F3tRGH47rKR93JqkBAADUZzbb9yorK0tvvfWWoqOj1bNnT73//vvKycmx1eUBALUsI69IY/63XSv2pkqSnAzS9Ds7a/pdEXJ2YicEAACA+s5gsVgsv/Ui3377rebMmaNvv/3W2ppgMBjk7u6ukSNH6rHHHtPQoUN/c7H1UUREhCQpMTHRzpUAaIgsFovyiozKyi9WVn6RMvOLlVVQpKz8Yi3YmayU7IuSJG83Z709Nlq3dAqxc8UAAAD4rar7HGqTwKBMZmamPv30U82dO1d79+4tvcGl/bhbtWql2NhYxcbGqnXr1ra6pcMjMABga8VGs84VFCszv0hZBaVBQFZ+sTILipSZ90sgkJVfpMyCYhUbzVe9XlN/D330aC91buZXS+8AAAAA15NdAoPyfvrpJ3388cf68ssvdf78+dKbGQwyGAwaNGiQxo8fr3vvvVfu7u7X4/YOg8AAQFUsFotyC42/zAC49KBvDQLKBQJZ+cXKuWi7RWgjW/jrf+N6KsTPw2bXBAAAgH3ZPTAoU1xcrKVLl2rOnDlat26dzObST7IMBoP8/f314IMP6rHHHlPPnj2vZxl1FoEB0DAVGU06V1CsrPxiZeT/8ol/1qWZAWXBQNalFoESk23/Ve1kkBp7uyvIx02BPm4K9HZXkI+7An3cSse83RXi56GIZn5yYr0CAACAeqXOBAblpaSkaN68eXr77bd19uzZ0gIutSx06dJFEydO1GOPPdagZh0QGAD1g8ViUc7Fkl9mAFx60P/1g3/mpRkBeYVGm9fg4+5y6eHfTYE+pWFAkI+79efASz8H+bgrwNOVIAAAAKCBqu5zaK3tiXXhwgV99913Wrt2rdLT02UwGGSxWFSWV+zdu1e/+93v9Oqrr+qdd97RyJEja6s0ALiswhJTxTUAyn/yX1CuFSC/SOcKimU02zZ/dXYyVHj4/+X7ijMBymYIeLo52/T+AAAAaNiue2CwZcsWzZkzR1999ZUKCgoklX4S5+/vr7Fjx+qRRx7Rvn379PHHH2v79u1KTU3VfffdpxUrVmjYsGHXuzwADYjZbFH2xZJKuwH8Oggoez2/yPazAHzdXRTkW/ap/6UAwNvt0ljFIMCfWQAAAACwo+vSklDWejB37lwdO3ZMkqwzCW6++WY98cQTuv/+++XhUXERrR9++EGPPvqoTp48qX79+mnz5s22Lq3OoSUBsI303ELtTcm57G4AZbsFnCsolsnGswBcnAzWqf5lD/9lQUDgpSAg6FIQ0NjbTR6uzAIAAACAfdV6S0JxcbGWLFmiOXPm6Pvvv5fZbLaGBCEhIRo3bpyeeOIJdejQ4YrXGDBggP79739r1KhR1m0ZAaAqX+w4pZeW7bPZwoB+Hi7WXv/ACosC/qolwNtdfp4u1rVYAAAAgPrEJoHBpEmTtGDBAmVnZ0sqnU3g5OSkYcOG6YknntCdd94pF5fq3apLly6SpLy8PFuUBqAeM5kt+vuKA/o47sRVj3NzdvrVg3/F3QHKLwbY2NtNbi5OtfQOAAAAgLrLJoHBBx98YP2+devWevzxx/X444+refPmNb6Wu7u7WrVqJScnfmEHcGV5hSWaPD9B6w+mW8fu69FCUS0DKs0E8HVnFgAAAABQUzZ5Knd1ddX999+vNWvW6Pjx4/rrX/96TWGBJLVq1UpJSUk6fvx4lcfu3LlTI0aMUEBAgLy9vdW3b18tXLiwRvc7c+aMJk+erM6dO8vb21shISHq37+/Pv30U5lMpsueYzab9fHHH6t///4KCAiQl5eXOnbsqMcee4yZEUAtSD53Qfe9v80aFrg4GTTj3q6aeX+kHu7bWsO6NFWvNo3VNshbfh6uhAUAAADANbDJDIMzZ84oMDDQFpeqtg0bNigmJkYeHh4aM2aMfH19tXjxYo0ePVrJycmaOnVqldc4fvy4+vTpo6ysLMXExOjOO+9Ubm6uli5dqnHjxmn9+vWaM2dOhXOKiop033336dtvv1W3bt0UGxsrd3d3nTp1SitXrtSrr74qX1/f6/W2gQZv98nzeurTXcrML5Yk+Xu66v2Hu+um9kF2rgwAAACoX67LLgnXm9FoVKdOnXT69Glt375dUVFRkqScnBz17t1bSUlJOnz4sFq3bn3V60yaNEnvv/++Zs2apcmTJ1vHs7OzFRkZqVOnTikpKanCdaZMmaJZs2bpH//4h/785z9XuJ7ZbJakGrVTsEsCUH3fJKTo2UV7VGws/WetbZC3Pnq0p9oF+9i5MgAAAMBxVPc51GYLBZw6dUqnTp1SUVFRlccWFhZaj78W69ev17FjxzR27FhrWCBJ/v7+euGFF1RcXKx58+ZVeZ2ytocRI0ZUGA8ICFD//v0lSZmZmdbxlJQUvfPOO7r55psrhQVSaVDA2guA7ZnNFv177SFNnp9gDQtubBeoJZNuIiwAAAAArhObPN2uXbtWbdu2VdeuXXXhwoUqj79w4YIiIiLUrl07bdy4scb3Kzvntttuq/RaTEyMJGnTpk1VXqdsR4aVK1dWGM/OzlZcXJxCQ0PVuXNn6/iiRYtkNBp1//33Ky8vT59//rlmzJihjz/+WCkpKTV+HwCqVlhi0h/mx+ut9UetYw/2bqlPxvdWgJebHSsDAAAA6jebrGHw1VdfyWKx6J577lGjRo2qPL5x48YaNWqUPvnkEy1YsECDBg2q0f2OHDkiSerQoUOl10JDQ+Xj42M95mqeffZZLV++XFOmTNHq1avVrVs36xoGXl5eWrJkiTw9Pa3H7969W1JpoBAWFqbU1FTra25ubvrHP/6hKVOm1Oi9ALiy9NxCPfnpbv2cnC1JMhikF0eEa3z/tixkCAAAAFxnNplhsG3bNhkMhst+4n8lZTMBtm3bVuP75eTkSCptQbgcPz8/6zFXExISom3btmnYsGFavXq1/vnPf+qDDz5QTk6Oxo0bp8jIyArHp6eXrsj+8ssvKzIyUomJicrNzdW3336roKAgPfPMM1q1atVV71lUVKTc3Fzrl9lslgMuIwFcd4lncnT3u3HWsMDbzVmzx/XUEze3IywAAAAAaoFNAoOkpCRJUseOHat9zg033CBJOnHihC1KuCZHjx5Vv379lJGRoc2bNysvL0/JycmaNm2aXn31VQ0ZMqTC1oplixo2adJEixcvVufOneXr66vbb79ds2fPliT961//uuo9Z8yYIX9/f+vXwYMHK6yTAEBat/+s7v9gm1JzCiVJzQM8tWjiTRoSHmLnygAAAICGwyYtCUajUZLk7Oxc7XPKji0sLKzx/cpmFlxpFkFubm61WiNiY2N18uRJHT9+XKGhoZIkHx8fPffcczp79qxmzZql+fPn66GHHqpw36FDh8rLy6vCtWJiYuTu7q5du3Zd9Z7PP/+8nnnmGevPffr04dNS4BKLxaIPNx/XjFUHVTbxJrpVgP73SE8F+7rbtzgAAACggbHJDIOgoNL9z8t2HaiOsmMbN25c4/uVrV1wuXUK0tLSlJ+ff9n1DcrLy8tTXFycwsPDrWFBeYMHD5YkxcfHW8fCwsIkle6i8GtOTk7y9fXVxYsXr3pfd3d3+fn5Wb+cnJwIDABJxUaz/rx4j15b+UtYcFdkM335ZF/CAgAAAMAObBIYlG1tuGDBgmqfM3/+fEm/7FRQEwMHDpRUujvDr61Zs6bCMVdSXFwsSVdsB8jIyJBU+oBf5pZbbpEk7d+//7LHZ2Zmqk2bNlVUD+DXzhcU65GPdmjhrtPWsSlDO+rNMVHycK3+zCUAAAAAtmOTwODuu++WxWLR119/ra+++qrK4xcuXKivv/5aBoNB99xzT43vN2TIELVr105ffPGFEhISrOM5OTl67bXX5ObmpnHjxlnHU1NTdfDgwQotDIGBgQoLC9OpU6es6w+Uyc7O1syZMyX9MtNAKg0hwsPD9f3332vdunXWcYvFohdeeEGS9MADD9T4/QAN2dH0fN3zXpx2nDgnSXJ3cdLbD0Zr8tAOzL4BAAAA7MhgscES/cXFxerUqZOSkpLk7OysyZMna/LkyWrZsmWF45KTk/Wf//xHb7/9tsxms1q2bKlDhw5V+BS/ujZs2KCYmBh5eHhozJgx8vX11eLFi3Xy5EnNnDlTU6dOtR4bGxurefPmac6cOYqNjbWOr1q1SnfddZeMRqOGDBmi6OhonT9/XsuWLVNGRoZGjRqlRYsWVbjvjh07dMstt6i4uFj33nuvWrRooS1btujHH39U9+7d9cMPP8jb27va7yMiIkKSlJiYWOM/A8DRbTmSqYmf71ZeYek6KEE+7vpwXA9Ft6p6DRIAAAAA16a6z6E2CQwkKSEhQQMGDFB+fr71U8FWrVqpadOmkko/5T916pSk0k/kfXx8tGnTJkVHR1/zPX/88Ue99NJL2rp1q0pKStS1a1c988wzGj16dIXjrhQYSNLOnTv1xhtvaMuWLcrIyJCHh4fCw8M1btw4TZw48bILOSYmJuqll17Sxo0blZubq1atWumBBx7QCy+8IB8fnxq9BwIDNFSfbT+pl5YlymQu/VdQp1BffRTbS80DPO1cGQAAAFC/1XpgIEkHDhzQww8/XGGhwLLwoPxtevTooU8//VSdOnWy1a0dFoEBGhqT2aK/rdivOXFJ1rGh4U00a0y0fNxtsnELAAAAgKuo7nOoTX87Dw8P1+7du7Vu3Tp9++23io+Pty4qGBQUpO7du+vOO+/UkCFDbHlbAA4ir7BET38Zrw2HMqxj/zegnf48rJOcnVivAAAAAKhLrsvHebfeeqtuvfXW63FpAA4q+dwFPTFvlw6dzZMkuTgZ9PeRXTS6Vys7VwYAAADgcpj/C+C6233ynP7vk93KKijdzjTAy1XvP9RDN7YPtHNlAAAAAK6EwADAdbU0PkV/WrRHxSazJKldkLc+iu2ltkHV300EAAAAQO27boGByWTS+fPndfHiRVW1rmKrVkxJBuobs9miWd8d1lvrj1rHbmofqPcf6iF/L1c7VgYAAACgOmwaGGRmZurtt9/W0qVLtX//fpnN5irPMRgMMhqNtiwDgJ1dLDbpj1/9rBV7U61jD/ZupVfujpCrs5MdKwMAAABQXTYLDLZu3ap7771XGRkZVc4oAFB/pecW6slPdunn0zmSJCeD9OLtnfV4vzbWbVYBAAAA1H02CQyysrJ09913KysrSz4+PnriiScUEBCg6dOny2AwaPbs2Tp37px27dqlZcuWqbCwUP369dP48eNtcXsAdcS+lBw9+ckupeYUSpK83Zz19tho3dIpxM6VAQAAAKgpmwQG77zzjrKysuTu7q5t27YpIiJCiYmJmj59uiTpsccesx6bmpqqsWPH6ocfftCNN96o119/3RYlALCzNYlp+n/zE3SxxCRJah7gqY9ie6pTqJ+dKwMAAABwLWzSTLxq1SoZDAY9/vjjioiIuOqxTZs21cqVK9W+fXvNnDlT69evt0UJAOzEYrHog03HNOGz3dawILpVgJb+rh9hAQAAAODAbBIYHD1augr60KFDrWPle5VNJlOF4z09PTVlypTSB40PPrBFCQDsoNho1p8W7dE/Vh1U2dIld0U205dP9lWwr7t9iwMAAADwm9ikJSE3N1eS1Lp1a+uYh4eH9fu8vDwFBARUOKdnz56SpB07dtiiBAC17FxBsSZ8tls/njhnHXvm1o76wy03sLghAAAAUA/YZIaBj4+PJFXYHrFx48bW75OSkiqdU1hYuihaenq6LUoAUIuOpudr5Htx1rDA3cVJ74yN1tNDOhAWAAAAAPWETQKDG264QZJ06tQp61hAQIBCQ0MlSRs2bKh0zpYtWyRJ3t7etigBQC3ZciRTI9+L08msC5KkYF93LXjqRt3RrZmdKwMAAABgSzYJDPr06SNJ2rlzZ4XxYcOGyWKx6J///KeOHDliHd++fbveeOMNGQwG9erVyxYlAKgFn20/qUfn/Ki8wtLZROFN/fTN7/opqmWAfQsDAAAAYHM2CQxiYmJksVj09ddfVxh/5pln5OLiovT0dEVERKhXr17q3Lmzbr75ZmVnZ0uSJk+ebIsSAFxHRpNZ05cl6i9L98lkLl3dcGh4iBZNuFHNAjztXB0AAACA68FmgcG4cePUt29fnThxwjrepUsXvf/++3J2dpbRaNTu3bt18OBB664J06dP17Bhw2xRAoDrJK+wRE98sktztyZZx54a0E7/faSHvN1tsm4qAAAAgDrIYLGUbYZ2/Rw6dEhz585VYmKijEajOnTooEceecS6U0JDFhERIUlKTEy0cyVAZcnnLmj8vJ06fDZfkuTiZNBrI7vqgV4t7VwZAAAAgGtV3efQWvl4MCwsTDNmzKiNWwGwkd0nz+n/PtmtrIJiSVKAl6vef6iHbmwfaOfKAAAAANQGmwQGr7zyiqTSxQ9jYmJscUkAdrQk/rT+vGivik1mSVK7YG99/GgvtQliVxMAAACgobBJYDB9+nQZDAYtWbLEFpcDYCdms0X/XndY72w4ah3rd0Og3hvbQ/5ernasDAAAAEBts0lgEBgYqHPnzqlVq1a2uBwAO7hYbNLUrxK0cm+adWxsn1Z6+a4IuTrbZH1UAAAAAA7EJk8BN9xwgyQpLS2tiiMB1EVncws1+n/brGGBk0Gadkdn/f2eLoQFAAAAQANlkyeB0aNHy2KxaOHChba4HIBatC8lR3e/E6c9p3MkST7uLvro0V56vH9bGQwGO1cHAAAAwF5sEhhMmjRJkZGR+uSTTzR37lxbXBJALViTmKb7P9imtNxCSVLzAE8tnniTBndqYufKAAAAANibTdYwSEtL0+zZszV+/HiNHz9eX3zxhcaOHatu3bqpUaNGcnZ2vur5rH0A1C6LxaIPNh3XP9cclMVSOtajdSP995EeCvJxt29xAAAAAOoEmwQGbdq0sU5dtlgs+v777/X9999X61yDwSCj0WiLMgBUQ7HRrBeW7NWi3aetY/dENdM/RnWTh+vVwz0AAAAADYdNAgOpNCi43PcA6o5zBcWa8Nlu/XjinHVs6q0d9ftbbmC9AgAAAAAV2CQwmDNnji0uA+A6Opqep8fn7tKpcxckSe4uTvr3A1G6vVtTO1cGAAAAoC6ySWDw6KOP2uIyAK6TzUcyNOnzn5RXWNr+E+zrrtnjeiqyZYB9CwMAAABQZ9msJQFA3fTp9pOavixRJnNpq1Dnpn6a/WhPNQvwtHNlAAAAAOoyAgOgnjKazPrbigOauzXJOnZr5xDNGh0lb3f+0QcAAABwdTw1APVQbmGJ/vBFvDYdzrCOPTWwnf4c00lOTixuCAAAAKBqNgkMHn/88Ws+12Aw6KOPPrJFGQAkJZ+7oMfn7tSR9HxJkquzQX8f2VUP9Gxp58oAAAAAOBKDxQZ7IDo5OV3TlmwWi0UGg0Emk+m3luCwIiIiJEmJiYl2rgT1wc6kc3rq0906V1AsSQrwctUHD/dQ33aBdq4MAAAAQF1R3edQm8wwaNWqVZWBQUFBgbKysqwhQVBQkLy8vGxxewCSvv7ptJ5bvFfFJrMkqX2wtz6O7aXWgd52rgwAAACAI7JJYJCUlFSt486fP68vv/xS06ZNU0BAgJYtW6awsDBblAA0WGazRf9ad0jvbjhmHbu5Q5DeGdtd/p6udqwMAAAAgCNzqs2bNWrUSJMmTVJcXJzS09M1fPhwnT9/vjZLAOqVi8Um/e6LnyqEBQ/3baWPY3sRFgAAAAD4TWo1MCgTFhamp59+WklJSfrXv/5ljxIAh3c2t1AP/HebVu1LkyQ5GaTpd3bWq3d3kauzXf7RBgAAAFCP2O2pYujQoZKkr7/+2l4lAA5rX0qO7n4nTntTciRJPu4u+ii2l2L7tb2mBUgBAAAA4NdssobBtfDx8ZEknTp1yl4lAA5p9b40TVmQoIslpbuLtGjkqY8e7aWwUF87VwYAAACgPrFbYBAfHy9JcnWlzxqoDovFog82Hdfrqw9ax3q0bqT/PtJDQT7udqwMAAAAQH1kl8DgxIkTmj59ugwGg6KiouxRAuBQiowmvfD1Pi3+6bR1bGR0c824t6s8XJ3tWBkAAACA+somgcEnn3xS5TFms1nnz5/Xrl279M033+jChQsyGAyaMGGCLUoA6q1zBcWa8Olu/Zh0zjr2x9s66neDb2C9AgAAAADXjU0Cg9jY2Bo9uFgsFknS008/rdGjR9uiBKBeOpqep8fn7tKpcxckSR6uTvr3A1Ea0bWpnSsDAAAAUN/ZrCWhLASoSkBAgAYMGKBJkybptttus9XtgXrnh8MZ+t0XPymv0ChJauLrrtmP9lS3FgH2LQwAAABAg2CTwODEiRNVHuPk5CRfX18FBATY4pZAvfbJtiS9vHy/TObSIC6imZ9mP9pTTf097VwZAAAAgIbCJoFB69atbXEZoMEzmsx69dv9mrftpHXsts4hmjUmSl5udtvUBAAAAEADxBMIUEfkFpbo91/E64fDGdaxiYPa69nbwuTkxOKGAAAAAGoXgQFQB5zKuqDx83bqSHq+JMnV2aDXRnbV/T1b2rkyAAAAAA2Vky0ucuLECd1yyy0aMmSIUlJSqjw+JSVFQ4YMqfbxQH22M+mc7nkvzhoWNPJy1Wfj+xAWAAAAALArmwQGn3zyiTZu3Kji4mI1b968yuObN28uo9GojRs36tNPP7VFCYBDWrz7tB76cIfOFRRLktoHe2vp7/qpT7tAO1cGAAAAoKGzSWDw/fffy2Aw6N577632Offee68sFovWrl1rixIAh2I2W/TP1Qc19aufVWwyS5Ju7hCkryf1U+tAbztXBwAAAAA2WsPgwIEDkqTu3btX+5yoqChJ0v79+21RAuBQXvl2v+ZuTbL+/Ejf1nrpzs5ycbZJhgcAAAAAv5lNAoOcnBxJUkBAQLXPKTv2/PnztigBcBjnC4r16fbSbROdDNJLd0bo0Zva2LcoAAAAAPgVm3yc6efnJ0nKysqq9jllx3p5edmiBMBhrN2fJpPZIkl6vF9bwgIAAAAAdZJNAoM2bdpIkjZu3FjtczZs2CBJatWqlS1KABzGir1p1u/viGxmx0oAAAAA4MpsEhgMHTpUFotF7777rlJTU6s8PiUlRe+++64MBoOGDh1qixIAh5B9oVhbj2ZKkpoHeCqyhb+dKwIAAACAy7NJYDBx4kS5uroqOztbQ4YM0Z49e6547M8//6yhQ4cqOztbLi4umjRpki1KABzC2v1nZbzUjjCia6gMBoOdKwIAAACAy7PJooetW7fW3//+d/3pT3/SoUOH1L17dw0aNEg333yzmjZtKklKTU3VDz/8oE2bNslischgMOjll19W+/btbVEC4BBW7v1lBs6Irk3tWAkAAAAAXJ1NAgNJ+uMf/6iLFy/q5Zdfltls1oYNG6zrFJRnsVjk5OSkl19+Wc8995ytbg/UeTkXShR3qR2hmb+HoloG2LcgAAAAALgKm276/te//lW7du3SmDFj5O/vL4vFUuHL399fDz30kHbv3q0XX3zRlrcG6rx1B86qxFTajjC8a1PaEQAAAADUaTabYVAmKipKX3zxhSwWi06cOKHMzNJPVIOCgtS2bVsektBg0Y4AAAAAwJHYPDAoYzAY1K5dO7Vr1+563QJwGDkXS7T5SIYkKdTPQ9G0IwAAAACo42zakgDg8r7bX74dIVROTsy0AQAAAFC32SQwyMnJ0SuvvKJXXnlFqampVR6fmppqPb6goOCa77tz506NGDFCAQEB8vb2Vt++fbVw4cIaXePMmTOaPHmyOnfuLG9vb4WEhKh///769NNPZTKZKh1vMBiu+BUbG3vN7wX126p9v/xzcTvtCAAAAAAcgE1aEj7//HNNnz5dHTp00LRp06o8PjQ0VJ9//rmOHj2q5s2ba/z48TW+54YNGxQTEyMPDw+NGTNGvr6+Wrx4sUaPHq3k5GRNnTq1ymscP35cffr0UVZWlmJiYnTnnXcqNzdXS5cu1bhx47R+/XrNmTOn0nmtW7e+bDgQFRVV4/eB+i+3sEQ/HC5dyyPEz13dWzWyc0UAAAAAUDWbBAarVq2SwWDQAw88UK3jDQaDxowZo1dffVXLly+vcWBgNBr15JNPysnJST/88IP1QX3atGnq3bu3XnjhBd13331q3br1Va8zc+ZMZWZmatasWZo8ebJ1fMaMGYqMjNTcuXM1ffr0Stdp06aNpk+fXqOa0XB9f+Csik1mSdLwLk1pRwAAAADgEGzSkpCQkCBJuummm6p9zo033ljh3JpYv369jh07prFjx1b4VN/f318vvPCCiouLNW/evCqvc/z4cUnSiBEjKowHBASof//+kmTd5QG4Viv2pFm/Z3cEAAAAAI7CJoFBenq6JKlp0+o/DIWGhkqSzp49W+P7bdy4UZJ02223VXotJiZGkrRp06Yqr9OlSxdJ0sqVKyuMZ2dnKy4uTqGhoercuXOl87Kzs/W///1Pr732mj744APt3bu3pm8BDUReYYl+uLQ7QhNfd/VsTTsCAAAAAMdgk5YEDw8P5efn68KFC9U+p+xYZ2fnGt/vyJEjkqQOHTpUei00NFQ+Pj7WY67m2Wef1fLlyzVlyhStXr1a3bp1s65h4OXlpSVLlsjT07PSeT///LOeeuqpCmPDhg3TvHnz1KRJk6ves6ioSEVFRdafzWazDAamqNdX6w+mq9hY1o7A7ggAAAAAHIdNZhiUzSzYtWtXtc8pO7ZspkFN5OTkSCptQbgcPz8/6zFXExISom3btmnYsGFavXq1/vnPf+qDDz5QTk6Oxo0bp8jIyErnTJ06VVu3blVmZqZyc3O1detWDR8+XKtXr9Ydd9xx2Z0VypsxY4b8/f2tXwcPHqTtoR5bseeX3RGG044AAAAAwIHYJDC4+eabZbFY9N5776mkpKTK40tKSvTee+/JYDBY1wqwh6NHj6pfv37KyMjQ5s2blZeXp+TkZE2bNk2vvvqqhgwZUikAmDlzpm688UYFBgbK19dXN954o7799lsNHDhQO3fu1DfffHPVez7//PPKycmxfnXq1ElBQUHX823CTvKLjNp4uLQdIcjHXb3aNLZzRQAAAABQfTYJDB577DFJpa0CY8eOvWprwoULF/Tggw/q8OHDFc6tibKZBVeaRZCbm3vF2QflxcbG6uTJk1q+fLn69+8vHx8ftWjRQs8995z+8Ic/aNu2bZo/f36V13FyctKTTz4pSYqLi7vqse7u7vLz87N+OTk50ZJQT/26HcGZdgQAAAAADsQmgcFNN92kMWPGyGKx6Ouvv1Z4eLj+9re/adOmTTp8+LAOHz6sTZs26dVXX1V4eLiWLFkig8Gg++67TwMHDqzx/crWLrjcOgVpaWnKz8+/7PoG5eXl5SkuLk7h4eGXbYsYPHiwJCk+Pr5aNZXNEigoKKjW8aj/VpZrR2B3BAAAAACOxiaLHkrSxx9/rMzMTH333Xc6ffq0XnrppcseZ7FYJEm33nprtbY+vJyBAwdqxowZWrt2rcaMGVPhtTVr1liPuZri4mJJV942MSOjdCq5u7t7tWrasWOHJKlNmzbVOh71W0GRURsOle4eEuTjpt5taUcAAAAA4FhsMsNAKt0pYc2aNZo1a5aaN28ui8Vy2a+WLVvqrbfe0urVq+Xh4XFN9xoyZIjatWunL774QgkJCdbxnJwcvfbaa3Jzc9O4ceOs46mpqTp48GCFFobAwECFhYXp1KlTmj17doXrZ2dna+bMmZJ+mWkgSXv37r3sGg1bt27V66+/LldXV91///3X9J5Qv2w4lK6iS+0IMRG0IwAAAABwPAZL2Uf+NmSxWJSQkKD4+HjrJ/hBQUHq3r27IiMjbdKzv2HDBsXExMjDw0NjxoyRr6+vFi9erJMnT2rmzJmaOnWq9djY2FjNmzdPc+bMUWxsrHV81apVuuuuu2Q0GjVkyBBFR0fr/PnzWrZsmTIyMjRq1CgtWrSownVWrFih/v37q2XLlnJ1dVViYqLWrl0rg8Ggd999VxMmTKjR+4iIiJAkJSYm/rY/ENQpkz7frZV70yRJXzzRRzfdwMKWAAAAAOqG6j6H2qwloTyDwaDo6GhFR0dfj8tLKv3kf8uWLXrppZe0YMEClZSUqGvXrnr99dc1evToal1j+PDh2rp1q9544w1t2bJFmzZtkoeHh8LDwzVt2jRNnDixwvF33323srOz9fPPP2vdunUqLi5WaGioxowZo//3//6fevfufT3eKhzMhWKj1h8sbUcI9KYdAQAAAIBjui4zDFB9zDCof1bsSdXvvvhJkvRg71aacW9XO1cEAAAAAL+w2wyDsnaEn3/+WZmZmbp48aKqyiSmTZtm6zIAu1m575fdEW5ndwQAAAAADsqmgcG8efP08ssv6+TJkzU6j8AA9cXFYpPWHyhtR2js7aa+7WhHAAAAAOCYbBYYvPjii/rHP/5R5WwCqXSNAzohUB9tPJSuiyUmSVJMRIhcnG22EQkAAAAA1CqbPM3s2LFDM2bMkCTdeuutSkhI0E8/lfZwGwwGmUwmZWRkWHclsFgs6t+/v1JTU2U2m21RAlAnrNj7SzvC8C60IwAAAABwXDYJDN5//31JUuvWrbVixQp169ZNrq6u1tcNBoMCAwMVExOjpUuX6t1339WWLVs0bNgwFRcX26IEwO4KS0zW3RECvFx1Y/tAO1cEAAAAANfOJoHB1q1bZTAY9PTTT8vFpeouh4kTJ2rUqFHas2eP3nvvPVuUANjdxkMZulB8qR2hc6hcaUcAAAAA4MBs8kSTmlo6DbtsawZJcnL65dIlJSWVznnkkUdksVi0YMECW5QA2N3Kcu0II7rRjgAAAADAsdkkMCgLBJo0aWId8/HxsX6fkZFR6ZwWLVpIko4ePWqLEgC7Kiwx6fsDZyVJ/p6uuol2BAAAAAAOziaBQXBwsCQpNzfXOhYSEiJnZ2dJ0oEDByqdUzYrIS8vzxYlAHb1w+EMFVxqR7itcwjtCAAAAAAcnk2easpaEQ4ePGgdc3Nzs45fru3g008/lSQ1a9bMFiUAdkU7AgAAAID6xiaBwc033yyLxaINGzZUGB89erQsFos+/vhjvfTSS0pMTNSPP/6oSZMmaeHChTIYDBo+fLgtSgDsprDEpO8OlO6O4Ofhon7tg+xcEQAAAAD8dgaLxWL5rRdJTExU165d5ePjo9OnT8vPz0+SdOHCBXXp0kVJSUkyGAwVzrFYLGrcuLESEhKs6xk0RGWzMBITE+1cCa7Vuv1n9eQnuyRJo7q30L8eiLRzRQAAAABwZdV9DrVZS8KGDRu0ZMkSGY1G67iXl5c2bNigfv36yWKxVPjq0qWLvv/++wYdFqB+WFWuHeH2bqF2rAQAAAAAbMfFVhcaOHDgZcdbt26tzZs369ChQ0pMTJTRaFSHDh0UHR1tq1sDdlNkNGnd/tLdEXw9XNTvBtoRAAAAANQPNgsMqhIWFqawsLDauh1QK7YcyVReUemsmlvDQ+Tu4mznigAAAADANtj7DfgNVpTfHaEruyMAAAAAqD8IDIBrVGw0/9KO4O6imzvSjgAAAACg/iAwAK5R3NFM5RWWtiMM7Uw7AgAAAID6hcAAuEbl2xGGd2F3BAAAAAD1C4EBcA2KjWatTUyTJPm4u2hAx2A7VwQAAAAAtkVgAFyDrccylXupHWFIeBN5uNKOAAAAAKB+ITAArsFKdkcAAAAAUM8RGAA1VGIya+2l3RG83Zw1kHYEAAAAAPUQgQFQQ9uOZSn7Qokk6ZbwENoRAAAAANRLBAZADZVvR7i9K7sjAAAAAKifCAyAGigxmbXm0u4IXm7OGhTWxM4VAQAAAMD1QWAA1MD241k6f6kdYXAndkcAAAAAUH8RGAA1sHJvmvX729kdAQAAAEA9RmAAVJOxXDuCp6uzBtOOAAAAAKAeIzAAqmnHiXM6V1AsSbqlUxN5utGOAAAAAKD+IjAAqmlFud0RhrM7AgAAAIB6jsAAqAaT2aI1+0rbETxcnXRLJ9oRAAAAANRvBAZANew4kaWsS+0Ig8OayMvNxc4VAQAAAMD1RWAAVMPKcu0II9gdAQAAAEADQGAAVMFktmj1vrOSJHcX2hEAAAAANAwEBkAVdiadU2Z+kSRpUFiwvN1pRwAAAABQ/xEYAFWgHQEAAABAQ0RgAFyFyWzRqku7I7i5OGlIeIidKwIAAACA2kFgAFzFrqRzysgrbUcY2DFYPrQjAAAAAGggCAyAqyibXSBJt9OOAAAAAKABITAArsBstmjVvtL1C9ycnTQknN0RAAAAADQcBAbAFew+dV5nc0vbEQZ0DJKvh6udKwIAAACA2kNgAFzBij3sjgAAAACg4SIwAC7DbLZoddnuCM5OGtqZ3REAAAAANCwEBsBlxCefV1puoSTp5g5B8qMdAQAAAEADQ2AAXMaKPb/sjjCcdgQAAAAADRCBAfAr5XdHcHU26FbaEQAAAAA0QAQGwK8knM5Wak5pO0L/G4Lk70k7AgAAAICGh8AA+JWV7I4AAAAAAAQGQHkWi0WrLu2O4OJEOwIAAACAhovAACgnITlbKdkXJUn9bghSgJebnSsCAAAAAPsgMADKKZtdIEm3044AAAAAoAEjMAAusVgsWnFp/QIXJ4Nui6AdAQAAAEDDRWAAXLLndI61HeHG9oG0IwAAAABo0AgMgEtW7vtldwTaEQAAAAA0dAQGgErbEVbuLQ0MnJ0Mui0i1M4VAQAAAIB9ERgAkval5Cr5XGk7wk3tA9XYm3YEAAAAAA0bgQEgacXeX9oRhnehHQEAAAAACAzQ4FksFq3a90s7Qgy7IwAAAAAAgQGQeCZXJ7MuSJL6tmusQB93O1cEAAAAAPZHYIAGb2W5doQR7I4AAAAAAJIIDNDAld8dwckgxbA7AgAAAABIIjBAA3cgNU9Jl9oR+rQNVBDtCAAAAAAgicAADVyFdoRutCMAAAAAQBmHDgx27typESNGKCAgQN7e3urbt68WLlxYo2ucOXNGkydPVufOneXt7a2QkBD1799fn376qUwmU5XnT5w4UQaDQQaDQWlpadf6VmAH5dsRDAaxOwIAAAAAlONi7wKu1YYNGxQTEyMPDw+NGTNGvr6+Wrx4sUaPHq3k5GRNnTq1ymscP35cffr0UVZWlmJiYnTnnXcqNzdXS5cu1bhx47R+/XrNmTPniuevW7dOH3zwgby9vVVQUGDLt4dacDAtT8czS/9/692msZr4eti5IgAAAACoOwwWi8Vi7yJqymg0qlOnTjp9+rS2b9+uqKgoSVJOTo569+6tpKQkHT58WK1bt77qdSZNmqT3339fs2bN0uTJk63j2dnZioyM1KlTp5SUlHTZ6+Tk5KhLly7q27evMjIytGnTJqWmpio0tGaL5kVEREiSEhMTa3Qefrt/rz2kt9YflSS9cneExt3Yxr4FAQAAAEAtqO5zqEO2JKxfv17Hjh3T2LFjrWGBJPn7++uFF15QcXGx5s2bV+V1jh8/LkkaMWJEhfGAgAD1799fkpSZmXnZcydPnqyLFy/q3XffvcZ3AXuyWCxaUa4dYRi7IwAAAABABQ4ZGGzcuFGSdNttt1V6LSYmRpK0adOmKq/TpUsXSdLKlSsrjGdnZysuLk6hoaHq3LlzpfOWL1+uefPm6e2331aTJk1qWj7qgMNn83Uso7QdoVfrxmriRzsCAAAAAJTnkGsYHDlyRJLUoUOHSq+FhobKx8fHeszVPPvss1q+fLmmTJmi1atXq1u3btY1DLy8vLRkyRJ5enpWOCcrK0tPPvmk7rnnHj344IO2eUOodSvK747QldkFAAAAAPBrDhkY5OTkSCptQbgcPz8/6zFXExISom3btunhhx/WqlWrtHr1akmSp6enJkyYoMjIyErnTJo0ScXFxXr//fevqfaioiIVFRVZfzabzTIYDNd0LVy7VeXaEYZ3ZTtFAAAAAPg1h2xJsJWjR4+qX79+ysjI0ObNm5WXl6fk5GRNmzZNr776qoYMGVJha8UFCxZo4cKFevPNN2u8uGGZGTNmyN/f3/p18ODBK66TgOvjyNk8HUnPlyT1bN1IIbQjAAAAAEAlDhkYlM0suNIsgtzc3CvOPigvNjZWJ0+e1PLly9W/f3/5+PioRYsWeu655/SHP/xB27Zt0/z58yVJ586d0+9+9zvdfvvteuSRR6659ueff145OTnWr06dOikoKOiar4eaK9+OMLwLswsAAAAA4HIcMjAoW7vgcusUpKWlKT8//7LrG5SXl5enuLg4hYeHX3a2wODBgyVJ8fHxkqRTp04pKytLK1askMFgqPBVtsBi06ZNZTAYlJCQcMX7uru7y8/Pz/rl5ORES0ItW1k+MGD9AgAAAAC4LIdcw2DgwIGaMWOG1q5dqzFjxlR4bc2aNdZjrqa4uFjSlbdNzMjIkFT6gC9JgYGBGj9+/GWPXbFihdLS0jR27Fh5enoqMDCw+m8Gtepoep4Ony1tR+jRupGa+ntWcQYAAAAANEwGi8VisXcRNWU0GhUWFqaUlBRt375dUVFRkkpbFHr37q2kpCQdOnRIbdq0kSSlpqYqJydHTZs2rdCq0KlTJx06dEgffvihnnjiCet4dna2brzxRh08eFDr1q3T0KFDr1rPoEGDtGnTJqWmptZ4bYOIiAhJUmJiYo3Ow7V56/sj+ve6w5Kkv97RWeP7t7VzRQAAAABQu6r7HOqQLQkuLi6aPXu2zGazBgwYoP/7v//T1KlTFRkZqcOHD+u1116zhgVS6boB4eHhWrJkSYXr/Oc//5GLi4uefPJJDR06VM8++6yeeOIJdezYUQcPHtSoUaOqDAvgWCq0I3ShHQEAAAAArsQhWxKk0jUGtmzZopdeekkLFixQSUmJunbtqtdff12jR4+u1jWGDx+urVu36o033tCWLVu0adMmeXh4KDw8XNOmTdPEiROv87tAbTqWka+DaXmSpOhWAWoWQDsCAAAAAFyJQ7Yk1Ce0JNSed9Yf0cy1pe0If7k9XE/c3M7OFQEAAABA7avXLQnAtVixN836/fCubKcIAAAAAFdDYIAG4URmgQ6k5kqSIlsGqDntCAAAAABwVQQGaBDKL3Z4e1cWOwQAAACAqhAYoEGouDsC7QgAAAAAUBUCA9R7J7MKlHjmUjtCC3+1bOxl54oAAAAAoO4jMEC9t6L87AIWOwQAAACAaiEwQL23qtzuCLcTGAAAAABAtRAYoF47lXVBe1NyJEldm9OOAAAAAADVRWCAem3lvvLtCOyOAAAAAADVRWCAeq3idoq0IwAAAABAdREYoN5KPndBe06XtiNENPNT60BvO1cEAAAAAI6DwAD11qpy7QgjmF0AAAAAADVCYIB6a0W53REIDAAAAACgZggMUC+dPn9BPydnS5LCm/qpbRDtCAAAAABQEwQGqJdW7/tldsHt7I4AAAAAADVGYIB6acVe1i8AAAAAgN+CwAD1zpnsi4o/lS1J6hTqq3bBPvYtCAAAAAAcEIEB6p2VzC4AAAAAgN+MwAD1zqp97I4AAAAAAL8VgQHqldSci9p98rwkKSzEVzc0oR0BAAAAAK4FgQHqlVV7f5ldMJzdEQAAAADgmhEYoF4pv37B7bQjAAAAAMA1IzBAvZGWU6hdl9oROjTxUYcQXztXBAAAAACOi8AA9cbqfeyOAAAAAAC2QmCAemPlXnZHAAAAAABbITBAvZCeW6idJ89JktoHe6tjCLsjAAAAAMBvQWCAemF1YposltLvb+/aVAaDwb4FAQAAAICDIzBAvbBiT7n1C7rRjgAAAAAAvxWBARxeel6hfkwqbUdoF+StMHZHAAAAAIDfjMAADm9N4llrO8II2hEAAAAAwCYIDODwVu5hO0UAAAAAsDUCAzi0zPwi7TiRJUlqG+St8Ka0IwAAAACALRAYwKGt3pcm86V2hOFdQmlHAAAAAAAbITCAQ1u1j3YEAAAAALgeCAzgsLLyi7TtWGk7QutAL0U087NzRQAAAABQfxAYwGGtSTxbrh2B3REAAAAAwJYIDOCwVu79pR3hdtoRAAAAAMCmCAzgkM4VFGvb8dJ2hJaNPdWlOe0IAAAAAGBLBAZwSGsT02S61I8woivtCAAAAABgawQGcEgryrUjjOhCOwIAAAAA2BqBARzO+YJibb20O0KLRp7q1sLfzhUBAAAAQP1DYACHs27/WdoRAAAAAOA6IzCAw6nQjsDuCAAAAABwXRAYwKFkXyhW3NFMSVLzAE9F0o4AAAAAANcFgQEcytr9Z2W81I4wvEso7QgAAAAAcJ0QGMChrCrfjtCNdgQAAAAAuF4IDOAwci6WaMuldoRm/h6Kbhlg34IAAAAAoB4jMIDDWLf/rEpMpe0Iw7qwOwIAAAAAXE8EBnAYK8u1I9zeLdSOlQAAAABA/UdgAIeQW1iizUcyJEmhfh6KbtnIzhUBAAAAQP1GYACH8F25doThXUPl5EQ7AgAAAABcTwQGcAjl2xFGdGV3BAAAAAC43ggMUOflFpboh8OluyOE+LmrRyvaEQAAAADgeiMwQJ23/kC6ik1mSdLwLk1pRwAAAACAWkBggDpvBe0IAAAAAFDrCAxQp+UVlmjT4dLdEYJ93dWjNe0IAAAAAFAbCAxQp60/mK5iY1k7QqicaUcAAAAAgFpBYIA6jd0RAAAAAMA+CAxQZxUUGbXxUGk7QpCPu3q1aWznigAAAACg4SAwQJ31/cF0FV1qRxjWJYR2BAAAAACoRQQGqLNW0Y4AAAAAAHZDYIA66UKxURsOpUuSAr3d1KdtoJ0rAgAAAICGhcAAddL6g+kqLCltR4hhdwQAAAAAqHUEBqiTyu+OcDvtCAAAAABQ6wgMUOdcKDZqw8HS3REae7upT1t2RwAAAACA2ubQgcHOnTs1YsQIBQQEyNvbW3379tXChQtrdI0zZ85o8uTJ6ty5s7y9vRUSEqL+/fvr008/lclkqnDs2bNn9fvf/159+vRRSEiI3N3d1aJFCw0ZMkRff/21LBaLLd9eg7XxUIYulpT+2cdEhMrF2aH/mgIAAACAQ3KxdwHXasOGDYqJiZGHh4fGjBkjX19fLV68WKNHj1ZycrKmTp1a5TWOHz+uPn36KCsrSzExMbrzzjuVm5urpUuXaty4cVq/fr3mzJljPT45OVmffPKJ+vbtq5EjR6px48ZKT0/X8uXLNWrUKD3xxBP68MMPr+fbbhBWVNgdIdSOlQAAAABAw2WwOODH4kajUZ06ddLp06e1fft2RUVFSZJycnLUu3dvJSUl6fDhw2rduvVVrzNp0iS9//77mjVrliZPnmwdz87OVmRkpE6dOqWkpCTrdUpKSuTk5CRnZ+cK18nLy1OfPn104MAB7du3TxEREdV+L2XHJiYmVvuc+uxisUndX12niyUmNfJy1c4XhzLDAAAAAABsqLrPoQ75JLZ+/XodO3ZMY8eOtYYFkuTv768XXnhBxcXFmjdvXpXXOX78uCRpxIgRFcYDAgLUv39/SVJmZqZ13NXVtVJYIEm+vr4aNmyYJOno0aM1fj/4xabD6bQjAAAAAEAd4JBPYxs3bpQk3XbbbZVei4mJkSRt2rSpyut06dJFkrRy5coK49nZ2YqLi1NoaKg6d+5c5XUKCwu1fv16GQyGGs0uQGUr9qZZvx/B7ggAAAAAYDcOuYbBkSNHJEkdOnSo9FpoaKh8fHysx1zNs88+q+XLl2vKlClavXq1unXrZl3DwMvLS0uWLJGnp2el89LT0/Xee+/JbDYrPT1dK1euVHJysl566SXdcMMNV71nUVGRioqKrD+bzWYZDIYqa20ICktM+v7AWUlSgJerbmwfaOeKAAAAAKDhcsjAICcnR1JpC8Ll+Pn5WY+5mpCQEG3btk0PP/ywVq1apdWrV0uSPD09NWHCBEVGRl72vPT0dL388svWn11dXfXGG29Ua6HFGTNmVDhXkoKDg6s8ryHYeChDF4pL2xFu6xwiV9oRAAAAAMBuGvQT2dGjR9WvXz9lZGRo8+bNysvLU3JysqZNm6ZXX31VQ4YMqbS1olTaymCxWGQ0GnXixAm9/PLLevHFFzVq1CgZjcar3vP5559XTk6O9atTp04KCgq6Xm/RoazaV353BNoRAAAAAMCeHHKGQdnMgivNIsjNzVWjRo2qvE5sbKxOnjyp48ePKzS0dPs+Hx8fPffcczp79qxmzZql+fPn66GHHrrs+c7OzmrTpo2ef/55ubi46E9/+pM+/PBDTZw48Yr3dHd3l7u7u/VnJ6cGndlYlbYjpEuS/D1d1e8GQhQAAAAAsCeHfFotW7vgcusUpKWlKT8//7LrG5SXl5enuLg4hYeHW8OC8gYPHixJio+Pr1ZNZQswli3IiJr54XCG8otKZ2fcSjsCAAAAANidQz6VDRw4UJK0du3aSq+tWbOmwjFXUlxcLKnitonlZWRkSFKF2QBXc+bMGUml6xmg5lbu/aUd4XbaEQAAAADA7hwyMBgyZIjatWunL774QgkJCdbxnJwcvfbaa3Jzc9O4ceOs46mpqTp48GCFFobAwECFhYXp1KlTmj17doXrZ2dna+bMmZJ+mWkgST///LNKSkoq1XPu3Dm98MILkqQRI0bY5D02JEVGk7671I7g5+FCOwIAAAAA1AEOuYaBi4uLZs+erZiYGA0YMEBjxoyRr6+vFi9erJMnT2rmzJlq06aN9fjnn39e8+bN05w5cxQbG2sd/89//qO77rpLTz75pObPn6/o6GidP39ey5YtU0ZGhkaNGqWhQ4dWOP7bb79Vv3791KpVK3l6eurkyZNasWKFCgoKdP/99+vBBx+sxT+J+mHz4cxy7QihcnNxyBwLAAAAAOoVhwwMpNJP/rds2aKXXnpJCxYsUElJibp27arXX39do0ePrtY1hg8frq1bt+qNN97Qli1btGnTJnl4eCg8PFzTpk2rtHjhI488IrPZrB07dmjDhg26ePGiAgMDNWDAAD366KPVvi8qKt+OMKJr5fUkAAAAAAC1z2CxWCz2LqIhi4iIkCQlJibauRL7KDKa1PPV75RXZJSvu4t2/XWo3F2c7V0WAAAAANRb1X0OZe437CruaKbyyu2OQFgAAAAAAHUDgQHsasWeNOv3I9gdAQAAAADqDAID2E2x0ax1+0sDAx93F/XvwO4IAAAAAFBXEBjAbuKOZiq3sLQdYWh4E3m40o4AAAAAAHUFgQHspuLuCLQjAAAAAEBdQmAAuygxmbV2/1lJkrebswZ0DLZzRQAAAACA8ggMYBdxRzOVc7FEkjQkPIR2BAAAAACoYwgMYBer9rI7AgAAAADUZQQGqHUlJrPWXNodwcvNWYPCaEcAAAAAgLqGwAC1btuxLGVfKG1HuKUTuyMAAAAAQF1EYIBaV353hNtpRwAAAACAOonAALXKaDJrTWJpO4Knq7MGhTWxc0UAAAAAgMshMECt2n78nM6XtSOEN5GnG+0IAAAAAFAXERigVq0o144wogvtCAAAAABQVxEYoNaUb0fwcHXS4E7sjgAAAAAAdRWBAWrNjyfO6VxBsaTS3RG83FzsXBEAAAAA4EoIDFBrKrQjsDsCAAAAANRpBAaoFSazxdqO4O7ipMHsjgAAAAAAdRqBAWrFjhNZyswvbUcYHNZE3u60IwAAAABAXUZggFqxam+a9fsR3WhHAAAAAIC6jsAA153JbNGqfb+0I9zSiXYEAAAAAKjrCAxw3e1MOqfM/CJJ0sCOwfKhHQEAAAAA6jwCA1x3K8vtjnA77QgAAAAA4BAIDHBdmcu1I7i5OGlIeIidKwIAAAAAVAeBAa6rXSfPKyOPdgQAAADg/7d352FV1vn/x18HhYOyKqJoY26576ZGgaFZ4JJLeZnkStNitl5qNeqo5DRppilN/copE8ndshx30WRR1FRAx7Es0HAZxdAExRQQ7t8ffDkjHVZFDsvzcV1el96fz7nP+8b7Ks/rfN6fG6hsCAxwV93ajjCgo5cNKwEAAAAAlAaBAe6a3HaE3MDAoQbtCAAAAABQmRAY4K6JO31ZF67ktiM83KqeXB3tbVwRAAAAAKCkCAxw12zO147A0xEAAAAAoDIhMMBdkZNjaOvR3Kcj2Ncw0Y4AAAAAAJUMgQHuivgzl5V85YYkqVdLT7nVoh0BAAAAACoTAgPcFVv+b3WBRDsCAAAAAFRGBAYoc7ntCLn7F9jXMOkx2hEAAAAAoNIhMECZO3w2VefSctsRfO6rJ7fatCMAAAAAQGVDYIAyt+XfPB0BAAAAACo7AgOUKcMwtPU/ufsX1LQzyb8d7QgAAAAAUBkRGKBMHTmbpv+mXpckPXRfPbnXdrBxRQAAAACA20FggDK15ej/2hEGdvSyYSUAAAAAgDtBYIAyYxiGJTCoYWeSfzsCAwAAAACorAgMUGaO/jdNZy//XztCCw/VcaIdAQAAAAAqKwIDlJnNR3k6AgAAAABUFQQGKBN/bEcIaE87AgAAAABUZgQGKBPHzl3Rmd9y2xEebO6hurQjAAAAAEClRmCAMkE7AgAAAABULQQGuGO3tiPYmST/9g1sXBEAAAAA4E4RGOCOHTt3Racu/S5J8m7uoXrOZhtXBAAAAAC4UwQGuGNb/0M7AgAAAABUNQQGuCO57QjJknLbEXg6AgAAAABUDQQGuCM/nr+qXy5ekyT1bFZXni60IwAAAABAVUBggDuy5ZanIwykHQEAAAAAqgwCA9y2W5+OYDJJAR1oRwAAAACAqoLAALftpwtXdTKvHaFpXdV3cbRxRQAAAACAskJggNu25d88HQEAAAAAqioCA9wWwzC0+ZZ2hP60IwAAAABAlUJggNuS8Gu6TqTktiP0aFJX9V1pRwAAAACAqoTAALdlc752BFYXAAAAAEBVQ2CA23Lr4xT7dWD/AgAAAACoaggMUGoJF64q4dd0SVL3JnXk5UY7AgAAAABUNQQGKLUtR5Mtv+fpCAAAAABQNREYoNRubUfoz/4FAAAAAFAlERigVBJ/TddPF65Kkrrd666GbrVsXBEAAAAA4G4gMECp3Lq6gHYEAAAAAKi6CAxQKvnbEQgMAAAAAKCqIjBAiZ1MSdfx5Nx2hC6N3XWPO+0IAAAAAFBVERigxG5dXTCQ1QUAAAAAUKVV6sDg4MGDGjBggNzd3eXk5CRvb2+tXbu2VOc4d+6cXn/9dbVr105OTk5q0KCBfH19tWzZMmVnZ+ebm5CQoNmzZ+vhhx9Wo0aN5ODgoMaNG2vs2LE6fvx4WV5ahXTr4xR5OgIAAAAAVG01bV3A7YqIiFBAQIAcHR0VGBgoFxcXrVu3TiNGjNCZM2c0efLkYs9x8uRJPfDAA7p06ZICAgI0aNAgXblyRevXr9fYsWO1a9cuhYaGWubPmDFDa9asUYcOHTRkyBC5urrq6NGjWrZsmb7++mtt27ZNDz/88N28bJtJunhNP5y/Iknq3Nhdf6pT28YVAQAAAADuJpNhGIatiyitmzdvqk2bNjp79qz279+vLl26SJLS0tLUs2dPJSUl6eeff1aTJk2KPM9LL72kTz/9VCEhIXr99dctx1NTU9W5c2edPn1aSUlJlvMsXbpUnTt3VteuXfOdZ/Xq1Xr66afVrl07HTt2rFTX0r59e0kq9evK2/+LSNS87T9Jkqb2b6Pxfi1sXBEAAAAA4HaU9HNopWxJ2LVrl06cOKGRI0dawgJJcnNz07Rp05SZmamwsLBiz3Py5ElJ0oABA/Idd3d3l6+vryTp4sWLluNBQUFWYYEkBQYGqlWrVvrhhx/yza9KeJwiAAAAAFQvlTIwiIyMlCT5+/tbjQUEBEiSoqKiij1Phw4dJElbtmzJdzw1NVUxMTHy8vJSu3btSlSTvb29JKlmzUrb5VGoU5eu6di53HaETn9yU+O6tCMAAAAAQFVXKT/dJiQkSJJatmxpNebl5SVnZ2fLnKK8+eab2rhxoyZOnKht27apU6dOlj0MateurW+//Va1ahX/6MADBw7o2LFj6tGjh9zd3Yucm5GRoYyMDMufc3JyZDKZin0PW7p1s0NWFwAAAABA9VApA4O0tDRJuS0IBXF1dbXMKUqDBg20b98+jR49Wlu3btW2bdskSbVq1dKLL76ozp07l6iWcePGyc7OTu+//36x8+fMmaNZs2blO+bp6Vns62zpYnqGatqZdDPH0IAOBAYAAAAAUB1UypaEspKYmCgfHx+lpKRo9+7dunr1qs6cOaOZM2fqnXfeUd++fa0erXir69ev64knntDx48f1zjvvqHfv3sW+59SpU5WWlmb51aZNG9WrV68Mr6rszXi8nWKnP6ZFo+/XvR60IwAAAABAdVApVxjkrSwobBXBlStXVKdOnWLPExQUpFOnTunkyZPy8vKSJDk7O2vKlCm6cOGCQkJCtHr1ao0aNcrqtTdu3NCQIUMUERGhqVOnatq0aSWq3Ww2y2w2W/5sZ1c5Mhu32vbq18HL1mUAAAAAAMpJ5fi0+gd5excUtE9BcnKy0tPTC9zf4FZXr15VTEyM2rZtawkLbtWnTx9JUnx8vNXY9evXNXjwYO3YsUNvvfWWZs+efTuXAQAAAABAhVUpAwM/Pz9JUnh4uNXY9u3b880pTGZmpiQV+hjElJQUScq3GkDKDQuGDBmiHTt26I033tDcuXNLVzwAAAAAAJVApQwM+vbtq+bNm2vlypU6fPiw5XhaWppmz54tBwcHjR071nL8/PnzOn78eL4WBg8PD7Vu3VqnT5/W4sWL850/NTVV8+fPl/S/lQbS/9oQduzYoUmTJmnevHl36QoBAAAAALAtk2EYhq2LuB0REREKCAiQo6OjAgMD5eLionXr1unUqVOaP3++Jk+ebJkbFBSksLAwhYaGKigoyHJ869atGjx4sG7evKm+ffuqa9euunz5sjZs2KCUlBQNGzZMX3/9tdV5vLy8NH78+ALrCgoKUtOmTUt8He3bt5ckHTt2rHQ/AAAAAAAAbkNJP4dWyk0Ppdxv/vfs2aPg4GCtWbNGWVlZ6tixo+bOnasRI0aU6Bz9+/fX3r17NW/ePO3Zs0dRUVFydHRU27ZtNXPmTE2YMCHf/KSkJEm5+yT88dGIeXr37l2qwAAAAAAAgIqo0q4wqCpYYQAAAAAAKE8l/RxaKfcwAAAAAAAAdxeBAQAAAAAAsEJgAAAAAAAArBAYAAAAAAAAKwQGAAAAAADACoEBAAAAAACwQmAAAAAAAACsEBgAAAAAAAArBAYAAAAAAMAKgQEAAAAAALBCYAAAAAAAAKwQGAAAAAAAACsEBgAAAAAAwAqBAQAAAAAAsEJgAAAAAAAArBAYAAAAAAAAKwQGAAAAAADACoEBAAAAAACwYjIMw7B1EdWZi4uLsrKy1KJFC1uXAgAAAACoBk6cOCF7e3tdvXq1yHmsMLAxJycn2dvb27qMYhmGoZSUFJEvoaLiHkVFxz2Kio57FBUd9ygqusp0j9rb28vJyanYeawwQIlcuXJFbm5uSktLk6urq63LAaxwj6Ki4x5FRcc9ioqOexQVXVW8R1lhAAAAAAAArBAYAAAAAAAAKwQGAAAAAADACoEBSsRsNis4OFhms9nWpQAF4h5FRcc9ioqOexQVHfcoKrqqeI+y6SEAAAAAALDCCgMAAAAAAGCFwAAAAAAAAFghMAAAAAAAAFYIDAAAAAAAgBUCAxTp4MGDGjBggNzd3eXk5CRvb2+tXbvW1mUB+u9//6uQkBD5+/vr3nvvlYODg7y8vDRs2DB9//33ti4PKNTcuXNlMplkMpm0f/9+W5cDSJK+/fZbPfbYY/Lw8JCjo6OaNWump59+WmfOnLF1aajmDMPQN998oz59+qhhw4aqXbu2WrdurfHjx+vkyZO2Lg/VyPLlyzV+/Hh1795dZrNZJpNJS5cuLXT+lStXNGnSJDVp0kRms1lNmzbVm2++qfT09PIrugzwlAQUKiIiQgEBAXJ0dFRgYKBcXFy0bt06nTp1SvPnz9fkyZNtXSKqsSlTpmju3Llq0aKFevfuLU9PTyUkJGj9+vUyDEMrV67UiBEjbF0mkM9//vMfde/eXTVr1tS1a9e0b98+eXt727osVGOGYejFF1/UZ599phYtWiggIEAuLi46d+6coqKitGLFCvn6+tq6TFRjkydP1oIFC9SwYUMNGTJErq6uOnLkiMLDw+Xs7Ky9e/eqQ4cOti4T1UDTpk116tQp1atXT05OTjp16pRCQ0MVFBRkNffatWvy9fXV4cOH5e/vr65duyo+Pl7h4eHq0aOHoqOj5ejoWP4XcTsMoABZWVlGixYtDLPZbMTHx1uOp6amGq1atTIcHByMpKQk2xWIam/dunVGZGSk1fHo6GjD3t7eqFOnjnHjxg0bVAYULDMz0+jWrZvxwAMPGKNHjzYkGfv27bN1WajmQkJCDEnGSy+9ZNy8edNqPCsrywZVAbnOnz9v2NnZGU2aNDFSU1PzjS1YsMCQZDzzzDM2qg7VzY4dOyyff+bMmWNIMkJDQwucO3PmTEOS8Ze//CXf8b/85S+GJGP27Nl3u9wyQ0sCCrRr1y6dOHFCI0eOVJcuXSzH3dzcNG3aNGVmZiosLMx2BaLae/LJJ+Xn52d1vFevXurTp48uX76so0eP2qAyoGDvvvuujh07piVLlqhGjRq2LgfQ9evXNWvWLDVv3lwffvhhgfdlzZo1bVAZkCspKUk5OTny8fGRm5tbvrHHH39ckpSSkmKL0lANPfroo2rSpEmx8wzD0OLFi+Xs7KwZM2bkG5sxY4acnZ21ePHiu1VmmSMwQIEiIyMlSf7+/lZjAQEBkqSoqKjyLAkoMXt7e0n8QxcVR1xcnN59910FBwerXbt2ti4HkCSFh4fr8uXLGjp0qLKzs/XNN9/ovffe06JFi5SYmGjr8gC1bNlSDg4OiomJ0ZUrV/KNbdq0SZLUt29fW5QGFCohIUHnzp2Tj4+PnJyc8o05OTnJx8dHJ0+erDR7xPCvaRQoISFBUu5/qP/Iy8tLzs7OljlARXL69Gnt3LlTDRs2VMeOHW1dDqCMjAyNHTtWXbp00VtvvWXrcgCL2NhYSVKNGjXUqVMn/fzzz5YxOzs7TZw4UfPnz7dVeYA8PDz03nvvafLkyWrTpk2+PQx27dqll156Sa+88oqtywTyKepzVN7x7du3KyEhQY0bNy7P0m4LgQEKlJaWJklWy7/yuLq6WuYAFUVWVpbGjBmjjIwMzZ07l2XfqBBmzpyphIQExcbGck+iQvn1118lSQsWLFC3bt104MABtW3bVvHx8XrhhRf0wQcfqEWLFpowYYKNK0V1NnHiRN1zzz167rnntGjRIstxX19fjRw5ktWEqHBK8jnq1nkVHS0JAKqEnJwcBQUFKTo6Ws8//7zGjBlj65IA7du3T/Pnz9f06dPZxRsVTk5OjiTJwcFB69evV48ePeTs7KxevXrpq6++kp2dnT744AMbV4nq7m9/+5tGjx6tadOm6cyZM7p69ap2796tGzduqHfv3tqwYYOtSwSqNAIDFCgvESss+bpy5UqhqRlQ3nJycvTnP/9ZK1eu1OjRo/N9AwHYys2bNzVu3Dh16tRJU6ZMsXU5gJW8/493795djRo1yjfWoUMHNW/eXCdOnFBqaqoNqgOknTt3Kjg4WK+88oqmTJmiP/3pT3J2dpavr682btwoe3t7HvONCqckn6NunVfRERigQHk9NwXtU5CcnKz09PRC+3KA8pSTk6NnnnlGYWFhevrpp7V06VLZ2fGfNtheenq6EhISdPjwYTk4OMhkMll+5T1l5sEHH5TJZNL69ettWyyqpdatW0uS3N3dCxzPO379+vVyqgjIb+vWrZKkPn36WI15eXmpTZs2SkxMVHp6enmXBhSqqM9Rtx6vLJ+laPpBgfz8/DRnzhyFh4crMDAw39j27dstcwBbygsLvvzyS40YMULLli2jRxwVhtls1rPPPlvgWHR0tBISEjR48GB5enqqadOm5VscoP99CPvxxx+txrKyspSYmCgnJyd5enqWd2mAJCkzM1NS4Y9OTElJkZ2dneXpSEBF0LJlSzVq1EgxMTG6du1aviclXLt2TTExMWrWrFml2PBQYoUBCtG3b181b95cK1eu1OHDhy3H09LSNHv2bDk4OGjs2LG2KxDVXl4bwpdffqnhw4dr+fLlhAWoUGrVqqXFixcX+Ouhhx6SJE2dOlWLFy9Wly5dbFssqqUWLVrI399fiYmJVs8Ef++995SamqonnniCTeVgMz4+PpJyN+b84/LuRYsW6ezZs3rwwQdlNpttUR5QIJPJpOeee07p6el655138o298847Sk9P1/PPP2+j6krPZBiGYesiUDFFREQoICBAjo6OCgwMlIuLi9atW6dTp05p/vz59IzBpt5++23NmjVLzs7Oev311wv8B+3QoUP5IIYKKSgoSGFhYdq3b5+8vb1tXQ6qsRMnTuihhx7Sr7/+qoEDB6pNmzaKj4/Xrl271KRJE+3fv19eXl62LhPVVHZ2th555BFFR0erfv36Gjx4sNzd3RUXF6ddu3apVq1aioyMVM+ePW1dKqqBxYsXa8+ePZKko0ePKi4uTj4+Prrvvvsk5T6547nnnpOUu5LAx8dHR44ckb+/v7p166a4uDiFh4erR48eioqKUq1atWx2LaVBYIAiHThwQMHBwdq7d6+ysrLUsWNHTZo0SSNGjLB1aajm8j5wFSU0NFRBQUHlUxBQCgQGqEjOnDmjmTNnatu2bbp06ZK8vLw0ePBgzZw5U/Xr17d1eajmMjIytHDhQq1du1Y//fSTMjMz1aBBA/Xp00fTpk1T27ZtbV0iqoni/u05btw4LV261PLntLQ0vf3221q3bp2Sk5PVsGFDDR8+XMHBwXJxcSmHissGgQEAAAAAALDCHgYAAAAAAMAKgQEAAAAAALBCYAAAAAAAAKwQGAAAAAAAACsEBgAAAAAAwAqBAQAAAAAAsEJgAAAAAAAArBAYAAAAAAAAKwQGAAAANrZ06VKZTCaZTCYlJSXZuhwAACQRGAAAAAAAgAIQGAAAAAAAACsmwzAMWxcBAAAAAAAqFlYYAAAAAAAAKwQGAAAAAADACoEBAADl5O2337bshC9JN27c0Lx589StWze5uLjIxcVFPXv21Mcff6ybN28Wep68c7z99ttFvl/v3r1lMpnUu3dvq7HIyEjLeSIjI2UYhr744gv5+vrKw8NDrq6u6tmzp5YtW5bvdZmZmVq0aJG8vb1Vt25dubi4yMfHR2vXri31z6Mg2dnZ+uSTT/TAAw/I1dVVbm5u6tatm+bPn6+MjAwlJSVZ6l66dGmprvlWf/y7KExaWprmzJkjHx8feXp6ysHBQQ0bNtSgQYP09ddfq6jOzj/+Pe3atUvDhw9X48aNZW9vr6ZNm1rmlvQpCdnZ2QoLC9Pjjz+uRo0ayWw2y8PDQ76+vlqwYIGuX79e5PXExsbq2WefVatWreTk5CRHR0c1btxY999/v15++WVt2LChyGsCAFQvNW1dAAAA1dGFCxfUr18/HT58ON/xgwcP6uDBgwoPD9f69etlZ3f3s/2srCwNGTJEGzdutKpl7NixOnTokD788ENdvnxZQ4cOVXR0dL55e/fu1d69e5WYmKhp06bddh3p6ekaMGCAdu/ene94fHy84uPjtWrVKi1evPi2z19a3333nUaMGKFLly7lO56cnKxNmzZp06ZNGjBggNasWSNnZ+ciz/XXv/5Vs2fPvqN6Tp8+rcGDB+vIkSP5jv/222+KiYlRTEyMPv30U23evFmtWrWyev3ChQv1xhtvKCcnJ9/xs2fP6uzZs4qLi9Mnn3yiq1evFns9AIDqgRUGAADYwJNPPqkffvhBr732mnbs2KHY2FitXLlSbdu2lSRt3LhRn3/+ebnUMmPGDG3cuFGjRo3S5s2bFRsbq1WrVql169aSpH/84x/auXOngoKCtHfvXk2YMEHh4eGKjY3VF198oUaNGkmSZs6cqWPHjt12HaNHj7aEBT179tSqVat06NAhbd68WcOHD1dcXJzGjx9/5xdcAjExMerfv78uXbqkBg0a6O9//7s2btyo2NhYbdy4UaNHj5YkbdmyRePGjSvyXN98841mz56tjh07asmSJTpw4ICioqI0adKkEtdz6dIl+fr66siRIzKbzXrllVf01Vdf6eDBg4qIiNDUqVNVu3ZtJSYmqn///kpLS8v3+n//+9+WsKBZs2b64IMP9N133yk+Pl7R0dH6/PPPNXLkSDk5OZX+hwUAqLoMAABQLoKDgw1JhiTD3t7eiIiIsJpz6dIlo0GDBoYko1OnTgWeJ+8cwcHBRb6fn5+fIcnw8/OzGouIiLCcR5IREhJiNef8+fOGi4uLIcnw9PQ0TCaT8e2331rNO3LkiGFnZ2dIMl577bUiayrMpk2bLLUMGDDAyMrKspoza9asfDWHhoZazSnqmm9169/FH2VmZhpNmzY1JBn9+vUzrl27VuA5PvvsM8s5wsPDrcZvrbVv377GjRs3Cq0nNDTUMveXX36xGh85cqQhyWjSpIlx8uTJAs8RFxdnODk5GZKMadOm5RubMWOGIclwcnIykpOTC60jNTXVyM7OLnQcAFC9sMIAAAAbePXVVwvss69bt66eeeYZSdLRo0etvim+Gx544AG9/vrrVse9vLz0xBNPSJJSUlL01FNPaejQoVbzOnXqJF9fX0myaicoqU8++USSZDab9fnnn6tmTeuuyenTp6tDhw63df7SWL16tZKSkuTo6Kgvv/xStWvXLnDe888/r549e0pSgfsp5LGzs9PixYtlNptvq56kpCStWbNGkvTxxx+rWbNmBc7r2rWrXn755QLrSU5OliS1atVKDRo0KPS93NzcyqUNBgBQOfB/BAAAbGDUqFGFjt1///2SJMMw9Msvv9z1WgIDAwsd69y5c6nmnTx5stTvn52drcjISEmSv7+/pcXhj+zs7Ipd/l8WNmzYIEny8/OTp6dnkXMffvhhSdK+ffsKnePj45Nvg8PS2rx5s7Kzs1W7dm3179+/RPWcO3dOp0+fthxv2LChJOmHH37QgQMHbrsWAED1wqaHAADYQJs2bQodq1u3ruX3V69eveu1FLRBXh53d/dSzbudek+cOKHff/9dktSjR48i5+Z9o383HTp0SJK0ffv2Yp+ikCfvG/yCdOrUqUzq+f333wtceVFUTffee68k6emnn9acOXOUkZEhHx8f9evXTwMHDpSvr6/at29f4usEAFQvrDAAAMAGClvmLinfkvDs7OwKU0tJ5v1xB/6S+O233yy/r1+/fpFzi1pOX1Z+/fXXUr+mqMcZ1qlT507Kua16JFlCGCk3oFq1apXq1KmjmzdvatOmTZowYYI6duyo+vXra8yYMbfdTgIAqLpYYQAAACqMivBNd15I079/f73//vt3fL4aNWqUST316tVTREREiV/3x70Ohg0bpkcffVRr1qzR9u3btXv3bqWkpOjixYtavny5li9frnHjxmnJkiXsYwAAkERgAABApWMymWQYRrHf5l+7dq2cKrozt34Df+HChSLnFjde0pUORf1sPDw8dO7cOWVmZpbLJovF8fDwkJTb7tG2bds7CiDc3Nz0wgsv6IUXXpAk/fjjj/rXv/6ljz76SOfOnVNYWJi6du1a4CaYAIDqh/gYAIBKxsXFRZJ0+fLlQucYhqHExMTyKumOtGjRQrVq1ZIkHTx4sMi5xY2X5GcjST///HOhY127dpWUu3dAZmZmkecpD3n1ZGRkWPYzKCtt27bVlClTtH//fjk5OUmS1q5dW6bvAQCovAgMAACoZPKWmhf14XHr1q1KTU0tp4ruTM2aNS2PmAwPD9f58+cLnJeTk6OwsLAiz5X3s/n5558L3YDx4sWL2rFjR6HnGDx4sCQpLS1NoaGhxZV/1w0aNMjSqhESEnJX3qNx48aWTS0vXrx4V94DAFD5EBgAAFDJ+Pn5SZK+//57xcTEWI0nJyfr1VdfLe+y7siECRMk5X6LPn78+AI3e5wzZ46OHj1a5HnyfjaZmZn66KOPrMazsrL03HPPFblJ4bhx49S4cWNJ0htvvKHo6Ogi33PPnj2Kiooqcs6daN26tYYPHy5JWr16tRYsWFDk/F9++UWrVq3Kd2z9+vVFBkhnzpzR8ePHJVnvfQAAqL4IDAAAqGReeOEF1axZU4ZhaNCgQQoJCdGhQ4e0d+9ezZs3T127dlVaWppatmxp61JLbNCgQRo0aJAkaePGjfLx8dGaNWsUFxenbdu2KTAwUNOnT1f37t2LPM/AgQPVpEkTSdKMGTM0adIk7dmzR4cOHVJYWJi8vb21YcMGeXt7F3oOs9mstWvXymw2Kz09XY888ohGjx6tr7/+WrGxsTp48KA2bNig4OBgderUSb169So2yLhTn376qZo3by5Jmjx5svz8/PTFF19o//79io+P186dO/XBBx/oscce03333ad169ble31ISIjuuecePfXUU1q0aJGioqJ0+PBhRUREaN68efLx8bGEKC+++OJdvRYAQOXBpocAAFQy7du31/vvv69Jkybp8uXLmjhxYr7xunXrav369ZoxY4YSEhJsVGXprVixQv3791dMTIy+//57BQYG5hvv2rWr/vnPf+r+++8v9BwODg5avny5+vXrp2vXrmnhwoVauHChZbxGjRoKCQnRb7/9pv379xd6Hm9vb0VGRuqpp57SmTNntGLFCq1YsaLQ+a6urqW40tKrW7euYmJi9NRTT2n37t2Kjo4ucuVDQfX8/vvv+uqrr/TVV18V+Bo7OzvNmjVLQ4cOLauyAQCVHCsMAACohCZOnKht27YpICBAderUkdlsVrNmzfTyyy8rPj5evXr1snWJpebi4qLIyEh99NFH6tGjh5ydneXi4qIuXbpozpw52rt3r+rWrVvseXx9fRUbG6sxY8aoUaNGsre3V8OGDTVs2DBFR0frtddeK1E93t7eSkhI0KJFizRw4EA1atRIDg4OcnR0VOPGjeXv7693331Xx48f19ixY+/08ovl5eWl6Ohobdq0SaNGjVLz5s1Vu3Zt2dvby9PTUw899JAmT56sqKgoLVmyJN9rV61apc8++0wjR45Uly5d5OXlpZo1a8rZ2Vnt27fXhAkTFB8fr+nTp9/16wAAVB4mwzAMWxcBAABQEklJSZYe+9DQUAUFBdm2IAAAqjBWGAAAAAAAACsEBgAAAAAAwAqBAQAAAAAAsEJgAAAAAAAArBAYAAAAAAAAKzwlAQAAAAAAWGGFAQAAAAAAsEJgAAAAAAAArBAYAAAAAAAAKwQGAAAAAADACoEBAAAAAACwQmAAAAAAAACsEBgAAAAAAAArBAYAAAAAAMAKgQEAAAAAALDy/wH4uGSGDmVQ8gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1200x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig = plt.figure(figsize=(12, 8))\n",
    "ax = plt.axes()\n",
    "\n",
    "ax.plot(np.arange(num_queries+1), results_setfit, label='SetFit')\n",
    "ax.legend()\n",
    "\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('num queries')\n",
    "\n",
    "display(plt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b6f8f75",
   "metadata": {},
   "source": [
    "<a id=\"nb3-part4-comparison\"></a>\n",
    "## IV. Comparing against BERT\n",
    "\n",
    "We perform the same steps again for BERT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0c64d809",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/cschroeder/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/utils/annotations.py:67: ExperimentalWarning: The function from_arrays is experimental and maybe subject to change soon.\n",
      "  warnings.warn(f'The {subject} {func_or_class.__name__} is experimental '\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "from small_text import TransformersDataset\n",
    "from transformers import AutoTokenizer\n",
    "\n",
    "\n",
    "transformer_model_name = 'bert-base-uncased'\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    transformer_model_name\n",
    ")\n",
    "\n",
    "num_classes = raw_dataset['train'].features['label'].num_classes\n",
    "\n",
    "target_labels = np.arange(num_classes)\n",
    "max_length = 128\n",
    "\n",
    "\n",
    "train = TransformersDataset.from_arrays(raw_dataset['train']['text'], \n",
    "                                        np.array(raw_dataset['train']['label']), \n",
    "                                        tokenizer,\n",
    "                                        target_labels=target_labels,\n",
    "                                        max_length=max_length)\n",
    "test = TransformersDataset.from_arrays(raw_dataset['test']['text'], \n",
    "                                       np.array(raw_dataset['test']['label']), \n",
    "                                       tokenizer,\n",
    "                                       target_labels=target_labels,\n",
    "                                       max_length=max_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b8b78fc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from small_text import TransformerBasedClassificationFactory, TransformerModelArguments\n",
    "\n",
    "setfit_model_args = TransformerModelArguments(transformer_model_name)\n",
    "clf_factory = TransformerBasedClassificationFactory(setfit_model_args, \n",
    "                                                    num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a2d93583",
   "metadata": {},
   "outputs": [],
   "source": [
    "from small_text import (\n",
    "    PoolBasedActiveLearner, \n",
    "    random_initialization_balanced,\n",
    "    BreakingTies,\n",
    "    SubsamplingQueryStrategy\n",
    ")\n",
    "\n",
    "# define a query strategy and initialize a pool-based active learner\n",
    "query_strategy = SubsamplingQueryStrategy(BreakingTies())\n",
    "active_learner = PoolBasedActiveLearner(clf_factory, query_strategy, train)\n",
    "\n",
    "\n",
    "# use the same initial set as above\n",
    "active_learner.initialize_data(initial_indices, train.y[initial_indices])\n",
    "labeled_indices = initial_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e4252f63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.85\n",
      "Test accuracy: 0.50\n",
      "---------------\n",
      "Iteration #0 (40 samples)\n",
      "Train accuracy: 0.53\n",
      "Test accuracy: 0.27\n",
      "---------------\n",
      "Iteration #1 (60 samples)\n",
      "Train accuracy: 0.90\n",
      "Test accuracy: 0.61\n",
      "---------------\n",
      "Iteration #2 (80 samples)\n",
      "Train accuracy: 0.95\n",
      "Test accuracy: 0.70\n",
      "---------------\n",
      "Iteration #3 (100 samples)\n",
      "Train accuracy: 0.96\n",
      "Test accuracy: 0.77\n",
      "---------------\n",
      "Iteration #4 (120 samples)\n",
      "Train accuracy: 0.98\n",
      "Test accuracy: 0.80\n",
      "---------------\n",
      "Iteration #5 (140 samples)\n",
      "Train accuracy: 0.96\n",
      "Test accuracy: 0.82\n",
      "---------------\n",
      "Iteration #6 (160 samples)\n",
      "Train accuracy: 0.97\n",
      "Test accuracy: 0.87\n",
      "---------------\n",
      "Iteration #7 (180 samples)\n",
      "Train accuracy: 0.97\n",
      "Test accuracy: 0.87\n",
      "---------------\n",
      "Iteration #8 (200 samples)\n",
      "Train accuracy: 0.95\n",
      "Test accuracy: 0.86\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[11], line 7\u001b[0m\n\u001b[1;32m      2\u001b[0m results_bert\u001b[38;5;241m.\u001b[39mappend(evaluate(active_learner, train[labeled_indices], test))\n\u001b[1;32m      5\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(num_queries):\n\u001b[1;32m      6\u001b[0m     \u001b[38;5;66;03m# ...where each iteration consists of labelling 20 samples\u001b[39;00m\n\u001b[0;32m----> 7\u001b[0m     q_indices \u001b[38;5;241m=\u001b[39m \u001b[43mactive_learner\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mquery\u001b[49m\u001b[43m(\u001b[49m\u001b[43mnum_samples\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m20\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m      9\u001b[0m     \u001b[38;5;66;03m# Simulate user interaction here. Replace this for real-world usage.\u001b[39;00m\n\u001b[1;32m     10\u001b[0m     y \u001b[38;5;241m=\u001b[39m train\u001b[38;5;241m.\u001b[39my[q_indices]\n",
      "File \u001b[0;32m~/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/active_learner.py:195\u001b[0m, in \u001b[0;36mPoolBasedActiveLearner.query\u001b[0;34m(self, num_samples, representation, query_strategy_kwargs)\u001b[0m\n\u001b[1;32m    192\u001b[0m indices \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39marange(size)\n\u001b[1;32m    194\u001b[0m representation \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdataset \u001b[38;5;28;01mif\u001b[39;00m representation \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01melse\u001b[39;00m representation\n\u001b[0;32m--> 195\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mindices_queried \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mquery_strategy\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mquery\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_clf\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    196\u001b[0m \u001b[43m                                                 \u001b[49m\u001b[43mrepresentation\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    197\u001b[0m \u001b[43m                                                 \u001b[49m\u001b[43mindices\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmask\u001b[49m\u001b[43m]\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    198\u001b[0m \u001b[43m                                                 \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mindices_labeled\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    199\u001b[0m \u001b[43m                                                 \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    200\u001b[0m \u001b[43m                                                 \u001b[49m\u001b[43mn\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnum_samples\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    201\u001b[0m \u001b[43m                                                 \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mquery_strategy_kwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    202\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mindices_queried\n",
      "File \u001b[0;32m~/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/query_strategies/strategies.py:218\u001b[0m, in \u001b[0;36mSubsamplingQueryStrategy.query\u001b[0;34m(self, clf, dataset, indices_unlabeled, indices_labeled, y, n)\u001b[0m\n\u001b[1;32m    214\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msubsample_size \u001b[38;5;241m>\u001b[39m indices_unlabeled\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m0\u001b[39m]:\n\u001b[1;32m    215\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbase_query_strategy\u001b[38;5;241m.\u001b[39mquery(clf, dataset, indices_unlabeled, indices_labeled,\n\u001b[1;32m    216\u001b[0m                                           y, n\u001b[38;5;241m=\u001b[39mn)\n\u001b[0;32m--> 218\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_subsample\u001b[49m\u001b[43m(\u001b[49m\u001b[43mclf\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdataset\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mindices_unlabeled\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mindices_labeled\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mn\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/query_strategies/strategies.py:231\u001b[0m, in \u001b[0;36mSubsamplingQueryStrategy._subsample\u001b[0;34m(self, clf, dataset, indices_unlabeled, indices_labeled, y, n)\u001b[0m\n\u001b[1;32m    227\u001b[0m subset_indices_unlabeled \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39marange(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msubsample_size)\n\u001b[1;32m    228\u001b[0m subset_indices_labeled \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39marange(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msubsample_size,\n\u001b[1;32m    229\u001b[0m                                    \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msubsample_size \u001b[38;5;241m+\u001b[39m indices_labeled\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m0\u001b[39m])\n\u001b[0;32m--> 231\u001b[0m indices \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbase_query_strategy\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mquery\u001b[49m\u001b[43m(\u001b[49m\u001b[43mclf\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    232\u001b[0m \u001b[43m                                         \u001b[49m\u001b[43msubset\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    233\u001b[0m \u001b[43m                                         \u001b[49m\u001b[43msubset_indices_unlabeled\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    234\u001b[0m \u001b[43m                                         \u001b[49m\u001b[43msubset_indices_labeled\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    235\u001b[0m \u001b[43m                                         \u001b[49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    236\u001b[0m \u001b[43m                                         \u001b[49m\u001b[43mn\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mn\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    238\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msubsampled_indices_ \u001b[38;5;241m=\u001b[39m indices\n\u001b[1;32m    240\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m np\u001b[38;5;241m.\u001b[39marray([subsampled_indices[i] \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m indices])\n",
      "File \u001b[0;32m~/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/query_strategies/strategies.py:78\u001b[0m, in \u001b[0;36mConfidenceBasedQueryStrategy.query\u001b[0;34m(self, clf, dataset, indices_unlabeled, indices_labeled, y, n)\u001b[0m\n\u001b[1;32m     75\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mquery\u001b[39m(\u001b[38;5;28mself\u001b[39m, clf, dataset, indices_unlabeled, indices_labeled, y, n\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m10\u001b[39m):\n\u001b[1;32m     76\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_validate_query_input(indices_unlabeled, n)\n\u001b[0;32m---> 78\u001b[0m     confidence \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mscore\u001b[49m\u001b[43m(\u001b[49m\u001b[43mclf\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdataset\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mindices_unlabeled\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mindices_labeled\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     80\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(indices_unlabeled) \u001b[38;5;241m==\u001b[39m n:\n\u001b[1;32m     81\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m np\u001b[38;5;241m.\u001b[39marray(indices_unlabeled)\n",
      "File \u001b[0;32m~/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/query_strategies/strategies.py:110\u001b[0m, in \u001b[0;36mConfidenceBasedQueryStrategy.score\u001b[0;34m(self, clf, dataset, indices_unlabeled, indices_labeled, y)\u001b[0m\n\u001b[1;32m     86\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mscore\u001b[39m(\u001b[38;5;28mself\u001b[39m, clf, dataset, indices_unlabeled, indices_labeled, y):\n\u001b[1;32m     87\u001b[0m     \u001b[38;5;124;03m\"\"\"Assigns a confidence score to each instance.\u001b[39;00m\n\u001b[1;32m     88\u001b[0m \n\u001b[1;32m     89\u001b[0m \u001b[38;5;124;03m    Parameters\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    107\u001b[0m \u001b[38;5;124;03m        subsequent methods do not need to differentiate maximization/minimization.\u001b[39;00m\n\u001b[1;32m    108\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 110\u001b[0m     confidence \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_confidence\u001b[49m\u001b[43m(\u001b[49m\u001b[43mclf\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdataset\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mindices_unlabeled\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mindices_labeled\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    111\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mscores_ \u001b[38;5;241m=\u001b[39m confidence\n\u001b[1;32m    112\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mlower_is_better:\n",
      "File \u001b[0;32m~/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/query_strategies/strategies.py:153\u001b[0m, in \u001b[0;36mBreakingTies.get_confidence\u001b[0;34m(self, clf, dataset, _indices_unlabeled, _indices_labeled, _y)\u001b[0m\n\u001b[1;32m    152\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget_confidence\u001b[39m(\u001b[38;5;28mself\u001b[39m, clf, dataset, _indices_unlabeled, _indices_labeled, _y):\n\u001b[0;32m--> 153\u001b[0m     proba \u001b[38;5;241m=\u001b[39m \u001b[43mclf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpredict_proba\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdataset\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    154\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m np\u001b[38;5;241m.\u001b[39mapply_along_axis(\u001b[38;5;28;01mlambda\u001b[39;00m x: \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_best_versus_second_best(x), \u001b[38;5;241m1\u001b[39m, proba)\n",
      "File \u001b[0;32m~/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/integrations/transformers/classifiers/classification.py:689\u001b[0m, in \u001b[0;36mTransformerBasedClassification.predict_proba\u001b[0;34m(self, dataset, dropout_sampling)\u001b[0m\n\u001b[1;32m    672\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mpredict_proba\u001b[39m(\u001b[38;5;28mself\u001b[39m, dataset, dropout_sampling\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m):\n\u001b[1;32m    673\u001b[0m     \u001b[38;5;124;03m\"\"\"Predicts the label distributions.\u001b[39;00m\n\u001b[1;32m    674\u001b[0m \n\u001b[1;32m    675\u001b[0m \u001b[38;5;124;03m    Parameters\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    687\u001b[0m \u001b[38;5;124;03m        If `dropout_sampling > 1` then the shape is (num_samples, dropour_samples, num_classes).\u001b[39;00m\n\u001b[1;32m    688\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 689\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpredict_proba\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdataset\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdropout_sampling\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdropout_sampling\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/integrations/pytorch/classifiers/base.py:141\u001b[0m, in \u001b[0;36mPytorchClassifier.predict_proba\u001b[0;34m(self, dataset, dropout_sampling)\u001b[0m\n\u001b[1;32m    139\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m torch\u001b[38;5;241m.\u001b[39mno_grad():\n\u001b[1;32m    140\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m dropout_sampling \u001b[38;5;241m<\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[0;32m--> 141\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_predict_proba\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdataset_iter\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlogits_transform\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    142\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    143\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_predict_proba_dropout_sampling(dataset_iter, logits_transform,\n\u001b[1;32m    144\u001b[0m                                                     dropout_samples\u001b[38;5;241m=\u001b[39mdropout_sampling)\n",
      "File \u001b[0;32m~/.local/share/virtualenvs/notebooks-R6UeU-dP/lib/python3.8/site-packages/small_text/integrations/transformers/classifiers/classification.py:698\u001b[0m, in \u001b[0;36mTransformerBasedClassification._predict_proba\u001b[0;34m(self, dataset_iter, logits_transform)\u001b[0m\n\u001b[1;32m    694\u001b[0m     text, masks \u001b[38;5;241m=\u001b[39m text\u001b[38;5;241m.\u001b[39mto(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdevice), masks\u001b[38;5;241m.\u001b[39mto(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdevice)\n\u001b[1;32m    695\u001b[0m     outputs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmodel(text, attention_mask\u001b[38;5;241m=\u001b[39mmasks)\n\u001b[1;32m    697\u001b[0m     predictions \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mappend(predictions,\n\u001b[0;32m--> 698\u001b[0m                             \u001b[43mlogits_transform\u001b[49m\u001b[43m(\u001b[49m\u001b[43moutputs\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlogits\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mto\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mcpu\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241m.\u001b[39mnumpy(),\n\u001b[1;32m    699\u001b[0m                             axis\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m0\u001b[39m)\n\u001b[1;32m    700\u001b[0m     \u001b[38;5;28;01mdel\u001b[39;00m text, masks\n\u001b[1;32m    701\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m predictions\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "results_bert = []\n",
    "results_bert.append(evaluate(active_learner, train[labeled_indices], test))\n",
    "\n",
    "    \n",
    "for i in range(num_queries):\n",
    "    # ...where each iteration consists of labelling 20 samples\n",
    "    q_indices = active_learner.query(num_samples=20)\n",
    "\n",
    "    # Simulate user interaction here. Replace this for real-world usage.\n",
    "    y = train.y[q_indices]\n",
    "\n",
    "    # Return the labels for the current query to the active learner.\n",
    "    active_learner.update(y)\n",
    "\n",
    "    labeled_indices = np.concatenate([q_indices, labeled_indices])\n",
    "    \n",
    "    print('---------------')\n",
    "    print('Iteration #{:d} ({} samples)'.format(i, len(labeled_indices)))\n",
    "    results_bert.append(evaluate(active_learner, train[labeled_indices], test))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b282d8bb",
   "metadata": {},
   "source": [
    "### Comparison\n",
    "\n",
    "This time we plot the learning curves of SetFit and BERT. Not only does SetFit run quite fast (albeit we did not measure the runtime here) but also its learning curve is quite steep.\n",
    "\n",
    "Do not draw any conclusions from this besides the impressive performance and runtime of SetFit. The comparison itself is not really fair and should be in favor of `bert-base-uncased` which has 12 layers and outputs embeddings of size 768 compared to `paraphrase-MiniLM-L3-v2` which has only three layers and outputs embeddings of size 384."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2c39b7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig = plt.figure(figsize=(12, 8))\n",
    "ax = plt.axes()\n",
    "\n",
    "ax.plot(np.arange(20, 240, 20), results_setfit, label=f'SetFit ({sentence_transformer_model_name})')\n",
    "ax.plot(np.arange(20, 240, 20), results_bert, label=f'BERT ({transformer_model_name})')\n",
    "ax.legend(loc='lower right')\n",
    "\n",
    "plt.xticks(np.arange(20, 240, 20))\n",
    "plt.ylim((0.25, 1.0))\n",
    "\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('number of instances')\n",
    "\n",
    "plt.title('AG\\'s News / Breaking Ties')\n",
    "\n",
    "\n",
    "display(plt)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}